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OPENING ADDRESS BY Dr. P.J. HILLERY, 
VICE-PRESIDENT, EUROPEAN COMMISSION . . 

I would like warml^ to welcome 'the delegates on 
behalf of the Commission. I ^ould especially like to 
thank the delegates and experts who have participated 
in the preparatory meet ings for this seminar and. those 
who have pr'e^^^ed the national review papers, the case 
studies arvd^he lead papers. 

We are witnessing an unprecedended expansion in 
vdcatiooal training facilities in all the Member States 
of ' the' CbTmnuriity^.-- ^ . 

In Britain, our host count-ry^ for this week, direct 
public expenditure on training has ^one . up from 
h 30 million in I968 to h 85 million in 197^,^ with the 
number of Government Training Centres rising from -13 
in 1963 to over 50 in 197^. ' ■ ■ " 

In France, the number of trainees helped by public 
expenditure has. almost doubl^ from 56O.OOO in 19'. "o 
920.000 in 1974. This explosive growth could just as 
easily be illustrated by figures from the other^Member 
States where direct public expenditure on- vocational 
training now runs at between one ami two per cent of 
the national budget . t 

Data on public expenditure is only the tip of the 
iceberg. To training provided or supported by the 
state we must add the greatly expanded , in-company trai- 
ning activities of industry and the spread of university 
courses with a direct vocational orientation. 

The importance of training springs from recognition^ 
of the necessity of an active employment policy. Several 
factors have contributed to this recognition. 

- Firstly, the pace of economic growth and develop- 
ment since the war has called' for a high-level of labour 
mobility - both^rof essional and geographical. It has 
been estimated that at least one worker in eight will 
change his profession in the next ten years and that 
over two milliori Europeans will leave the land oyer the 
same period . 1 y 
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Secondly^ there has been a recognition of the right 
of every worker > and of- every category of worker to be 
trained and given employment X"o th^ full extent of ^his 
or- her own capacity and talent. This is exemplified in 
Germany^ s employment promotion -ftct of 1971 or the French 
law on the'organisation of permanent vocational training. 
These give each individual the statutory ri'ght to under- 
take tfrainipg^courses with the assistance of state grants. 

The growing participation of women in employment 
has called for a system of trainirlg suited 'to their 
particular' needs , It is recognisecl in all the member 
states that much htxs stil-1 to be - done to improve the 
employment position and" opportunities of certai-n groups - 
in partiQular women, migrant workers and the, han(*i^apped , 

As Commissioner Tor Social Affairs I have initiated 
Community measurjes to encourage the member stateS' to 
tackle the most urgent problems facing these underprivi- 
.•leged groups. Clearly effective programmes of vocational 
. training^ will also have a very important role in' these^ 
developments . - ' ■ ' ■ 

Thirdly, there ^has -been an increaaed understanding 
of the significance of ' vocational training in fulfilling . 
other economic and social goals. For' example the 
achievement of planned regional development calls for 
carefully thought-out vocational training in .order to 
'ad75pt the manpower , and womanpower of the regions tq the 
needs of industries and enterprises. Again, the high 
•levels of employment, which we have enjoyed until , 
recently, entailed high levels^of inflation, caused t^p 
someex^ent by s.^ecific manpower . shortages . An active 

^power policy, involving vocational retraining, has. 
Contributed substantially to reducil^^-inflat ion while 
still '^enabling- a high level of employment to. be enjoyed . 
Faced as we- are with growing inflation, unemployment and 
the structural changes caused by the rise in. the price 
of energy and raw materials, the need -Tor an active, anq 
vigorous \qianpower policy becomes even mQr.e_ crucial . 
Vocational retraining is -OQt a pass-time for tlfie unem- 
ployed ; it is a pr.e requisite for continued ^economic 
development .' 

The Commission of the European Communities has ^^ept 
^apace with and sought to initiate, supplement and extend 
* vocational training.*- • ^ ' 

Through the-» European Social Fund it has been .intima- 
tely connected with the training effort in the differerit 
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member stale:-., with the defining of common traini/g^eeds 
and with the coordination of national training policies. 
The growth of the Fund has been remarkable to the point 
where it now has an annual budget of some ^150 m. and 
where in 1973 Social Fui»d monies going to Britain for 
example, represented .of direct spending on'voca- 

tional. training by the Department of Employment in 
Britain. Ttie introduction of'^^the Regional Fund Jf'ill 
give -the Commission a further instrument in comj^atting 
employment . problems . - ' 

The Sacial Fund gives special emphasis to the trai- 
ning of workers from less developed regions, to agri-cul- 
tural and textil.fe' workers .,• and to programmes for the^ 
handicapped and migrant workers. The overriding policy- 
aim however is to assist genuinely promotional and 
innovative pj^ogrammes as well as raising the standard 
of training provided. 

In this context I am pleased ta refer to the recent 
decision of the Council of Ministers to es^tablish the^ 
European, Vocational. Centre as proposed by the Commission. 
The role* of this centre will be to coordinate research 
into various aspects of training and darry out its own '-..^^ 
specific projects. It 'will also organize courses and 
seminars' and will establish a selective documentation 
centre on vocational training. The centre will be 
imanaged by. a Committee composed of representatives of 
the Commission, the Member States and the -.Trade-Unions 
and employers. This will be the first time the social 
partners will be involved in the active management of^^a ' 
European Institution. This is a very welcome develop^f 
ment.. ^ ■/ \ 

This seminar on the Evaluation of Vocational^ 
Training comes at, a very opportune time*. It provides an 
^opportunity for senior training officials from the member 
countries to come together to exchange their ideas and 
experiences, to identify common problems and to^work 
together^ 1:0 resolve them.- Vocational training is now ^ 
serving a' wide variety of goals and as a result its 
evaluation has become increa^ngly complex. Parhaps as 
result of 'th^ variety of bodices concerned, and of the 
rapid growth in trainings-programmes have not always been 
'well thought out or have not achieved the results hoped 
.for. Expansion may have leapt ahead of the application 
of techniques of evaluation. e - .. . 

The task of the seminar is' to make, this complexity 

8" 
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manaf^onb 1 < ■ fv>r' rMMMci.'Ms and otherr, who muGt'take 
decisionr. about th^= ncile, dor.i p-.n .and imt>lemeiitat ion ' of 
Xr-aininfT; prop;rarnmr-s , and who nmt.;t; , ♦nu;ur(..^ tHo quality of 
l\vs) traininr, , 

^ ■ Tbe r.eminar will ben;in te^ilh an overall -view of t^'ie 
dc-Ve lopme^rit of t. h'.? -r-oo i "lI and economic role of training 

■"the participant-o wi-11 then discOss papers dealing w^ith 
th^: uses and problorns rf coc.l-benef i't techniques as 
•applied, to 'tI^aiIli^r prop,ratnines . The measurement of non- 
monat^ary. vririat)lo'J-V;, • such as the impact of training on 
adaptabi lit;/ , and the coni'idence of trainees, are also 
consldei'od in a selfees of papers. Various techniques of 
p;athorinn; infoi^mati^p abovit tht:^ outcome of specific 
training coursers , espec ially on their impacts on the long 
t;erm career opportunities, are pr'esented f or discussion. 
This is -in keeping with the emphasis on training as a 
stage, or a series of stages, in the career of each 

The issues which you will be discussing in the 
coming week are of great importance. All of us who are 
involved in vocational training have t he - re spon s ib i 1 it y 
and the challenge of ensuring that the training carried 
out in the Community is effective, in that it takes- 
account*- of the potentia.i and the' capabilities of the . • 
trainees, l:hat it uses the 'best techniques for. teaching 
and -that the trarining provided is matched to job-opp'or- 
tunities. Finally", if this trainihg is to be delivered 
to as* many workers as possible it must be cost-effective 

The Commission intends to 'develop its work in the 
area of vocational training and I am confident that the 
results of this seminar will be very significant for the 
development of techniques of evaluation. I will be 
following with great interest th'ie' 'proceedings and. the 
recom'mendat ions of this seminar, ^ ^ 

I wish yoi^,well in your work. - ' , 
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INTRODUCTION 



an instrument of economic policy [^^enenally and . 
employment pol»icy in particular vocational training has 
come to play an increasingly important role in th-o member 
states of the Community. ^ . . 

Alongside with the f^rov;th and expansJ^on in ,training 
outlined in the o.pen inf^ npe.'.^'c h of Dr. Hillery, most 
countrujc, have imp lemeri't ed substantial and far-reaching 
•reorganisation ol-* their systems of vocational training' 
in recent years. This was In part a response to the 
fact that'as'a rr^sult of the variety of bodies concerned, 
and of the rani J gr'owth in training, programmes l:ave not 
always \ice^ tiioufrht out or have not achieved the 

results hoped for. " 

Thesf ' do ve lopment s ha". /d to the demand for more 
adoquatf-' s.yst,emr> and conce{V^-- of evaluation o.f vocational 
training. ' T\\^j evaluation of vocational training is not 
a new science, but rather involves an attempt to ( i ) ^ map 
out and lo r)rganise' th-e 'different elements- involved in^^ 
the exercise of evaluating a given training programme ( ii ) 
to apply the findings^ and concepts of relevant sciences 
e .g. ' economics , organisational theory, psychology of 
learning etc . 

Lwaluat ion • can be used (i) to help in the control 
and management of what has a substan.tial 'and compl'ex 
act ivity ^( ii) to provide information for decisions on the 
future scale and orientation of training. Its signifi- 
cance ^as become particularly crucial in the light of the 
current employment crisis. The numbe*r of persons unem- 
ployed in the Member Stat.es of the Community doubled in 
/l97^ and has been continually increasing throughout most 
'of 1975. This crisis has lead to a searching reappraisal' 
of employment policies and of the appropriate str#:egy 
and role for training. Should additional public expeYidi- 
ture be devoted to training or rather to job-creation ? 
V/hat sort of training can be given to young persons in 
order to improve their chances of access to' employment ? 

How important is the placement rate in the assessr 
ment of the success of a training programme ? ■ 

Unfortunately neither the policy makers of the 
Member States, ^hor ,of training organisations within mem- 
ber states are 'in a position to begin to answer questions 
of this kind. '^^The explanation for this unsatisfactory 
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c^^t^t'e of aiM'aitT) can 'be found, in the rapid growth of 
training, tYy^). I'ccMjnt birth. of new training p^^ganisat ion , 
and the inherent difficulty In conceptualisy and 
drganisating evaluation systems. 

The Commis-sion of the European Communities has been 
closely watching the development of this situation parti 
cularly through its experience with the European Social 
• Fund. In Autumn 197^1 it requested a number o-f brief 
reports on the situation as regards evaluation in the 
Member States vQ^^^^ Community. A fairly consistend 
picture emerged of * • 

(i) recognition of the importance of evaluation 

I (ii)- extreme practical difficulty in implementation 

(iii^ a number of initiatives in the field, usually 
consequent upon . reorganisation of vocational 
training systems. 

.Some -extract s from the reportts are : 

"The-. b.f^lance sheet^ which can be currently established is 
fairly thin" (France) 

"With regard to routine training int^g'fid^^d -f or young 
people it is- even less opp-ort une>^ specify that, there . 
is no c la^ssif iecl^ con>t^6T sy^s-fc-dfT^ especially as there 
is no specific init ial plan " (Italy.) 

"'Thanks to the Training Services Agency there is , for 
the first time in the United .Kingdom, the means to pro- 
vide an^ overall view and to define a gli^bal strategy 
for the vocational training system. . . The rationali- 
sation of budgetary choice set* up by the Agency in the 
preparation of its objectives will allow the establish- 
^ ment of evaluation criteria" (United Kingdom) 

"It is difficult tlb forrij a complete picture of the whole 
area of further vocational training. 

"There is no methodological apparatus by which the value 
of training schemes within undertakings can be gauged. 

"The information available is a little ,out of date." • 
(Germany) 

As part of its role in^^st imulat ing the interchange 
of ideas on common problems" arid encouraging the spread of 
best practices , the Directorate -General for Social 

11 - 
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Affairs organlr^^Ml a seminar on t^k^^^subj ect in, Manchester 
V University in.^J.anuary 1975- The sefninar participants 
were some 60 ti^aining expert's with responsibil:|,t ies for 
. t^he activation and management of continual vocational 
.\ training mea'sures (or, if such be the case, research in 
the field of training) in private or public institutions, 
in companies and in employers' and employees' organisa-. 
tions, representing all 9 mer^i"ber-state>s . 

The. Seminar programme X^is designed, to provide 
maximum opportunity fbr allHRarticipants to contribute 
ideas -and inf QrmaVion\based dh their -own experience. 
After an initial' s^siNon in which experts present'ed 
.papers in plenary sesasion , the seminar divided into 
small working groups ^nj order to discuss .the issues 
which had been raise^i^nd to produce their own reports 
and recommendations. To facilitate productive discus- 
sion, the. p^roups were assigned to one of, three topic 



areas : 



ii) The evaluation of training programmer at nat: 
nal or sectoral level, including the conside- 
ratio^is which should inform*^ training policy 
decisions -and the role of cost-benefit analysis; 

(ii) The evaluation of the. training scheme itself , 
^ e.g. methods of select ion-, course design and 

planning, teaching methods and control 
meas-ures 

(iii) The e\^luation of employment outcomes with 
■ , respect to "tPi^ impact on-, the labour market as 
a whole^afid on th^ particular individuals who 
receive training. 

Appropriate case st udies were provided for each 
working group to serve as a focus for ciiscussion,. Group 
repgrts were not necessarily tied to the problems presen- 
ted in their' case studies. • - , 

There followed plenary sessions in which, the group's 
reports were presented and discussed, tv^a separate groups 
reporting on -each topic .area so .that differing conside- 
rations could be developped and compared.' 

The present volume is based on th^ working papers 
andj! the results of the discussions held at the ^seminar . ^• 

This volume is in five chapters. 

; ' ^ O 



Chapiter 1 describes and defines the different levels of 
analysis which can be , used in evaluation. 
.They-are : analysis in the domains of socio- 
economic policy , 'manpower- policy," trainir};g 

^ polic:^/", evaluation within the training pro.- 

* ^ gramme i-tself. ' ' ^ , 

Chapter. 2 deals with. e;valWtio-n within the-A^^rious v 
d'om^ii7,s ise,^' the evaluation . of training pro- 
grammes. 

Chapter 3 covera evaluation within the ^^aining program- 
me e*. g. the assessment ' of' trainees and of \ ^ 
teaching methods. / 

Chapter 4 discusses Cost-Benefit Analysis • techniques 'as 
. , applied to- Vocational Training. 

finally ' .. 

ChapterV3 presents an Overview of Evaluation andr its 
p Practical Implementation^.' 

A number of. papers follow the overview given in 
each chapter. Their aim is to give practical examples 
of the more theoretical • discussions in the. text. Some- 
times- however they develop in greater detail, or present 
under a d^.fferent form some of the ideas contained in 
th^ main text . ' 
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CHAPTER 1 



THE DOMAINS OF EVALUATION 



In this chapter we shall consider the various 
levels of analysis into which different aspects of the* 
•training system may be ordered. Tiae problems of eva- - 
luation would seem to d:i.vide * ipto : ' • 

1.. Methods of assessing learning * . 

2. Methods of assessing the effect of learning' to" 
terms of tas^ performance * ^ 

-3. Methods of ass-essing the empl^/rrJ^nt outcome for 
the individual ' ' . 

■ 4. Methods df assessing' the impact on the employ^:^ — 
ment situation as a whole - 

5. Methods of assessing the broad socio-economic 
outcome. 

Thus atT the most specif ic^ level are the particular 
teaching/learning systems through which skills are ^ 
developed. Here a considerable body of systematic tech- 
nique has been developed during the last thirty years, 
and the process of evaluation must reflect this. Beyond 
the immediate learning situation are the worlds of work 
and employment out of which training activities ultima- 
tely flow. , While recognising the integration of these 
different levels, it is helpful to. distinguish between 
them at least initially in order to keep ttoe practical 
task of evaluation within reasonable bounds. 

The" accompanying diagram (fig.l) represents the 
field as a series of embedded elements : the system/for 
controlling a training programme is embedded within the 
domain of training policy ; the system for controlling 
training policy is embedded within the domain of man- 
power policy ; and the manpower policy system is embedded 
within the domain of socio-econbmic policy. Evaluation 
can thus be sfeen as the operation of th% va'rlous feedback 
loops (the diagram has been restricted to showing only 
the principal, formal loops). 
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Gene rail:/ :-.:pcakvlii[-^^, evaluation follows a time se- 
quence across the uifrerent domains.^ V/ithin the inner- 
most domain, feedback on the effectiveness of learning 
operates quickly - v;ithin days or a fev;- weeks - while 
the feedback of information concerning employment out- 
comes may well take months as a minimum/ .At th^ level 
%f the .manpov;er domain the i;;el*evant informat'i'on may not 
become availablecf'or years, a^d this represents a severe 
limitation on research in this area. There is also a 
clear tendency for the feedback data to be , har^der to 
define and to obtain tis the larger domains ar^e appro^- 

the dimensions of'relevant outcomes 
in. the internal evaluation of a- 
training programme since trie situation is more confined. • 
But as outcomes develop into the larger, domains the/ 
interact with more 'Variables o\*i'- a larger period -of 
^ime. -^The "result is that, the evaluation researcher is 
wricwtain about, wha/u date to pursue. He ^r&\less able to 
neek, answer-s to pre-determined quesyons anoYmore 
constrained t^wonder -what questions he Ahould be asking, 
frhe difficultv. shows clearly in the units of ev^uation 
whic^ ar-e app^'opriate to each domain. These may \be 
represented hierarchically as : ^ , •• 



ched. By comparisis 
a^e easier to predict 



1 . factors in life 

2. factors in work 

5. factors in. careers 

4. factor-s in employ- 
ment 



\ )Socio- 
) 

) Man- ) 



Trai- 



o . 



factors 
ope: 



) ning 
policy 



factors in skills) Trai 
) ning 
in-. ) prog 

at ror^s 

Typically, the various domain 
indicated co n c e r n e d 



pow^r 



')Economic 
) 

policy^ policy 

) - 




of control have as 
themselves with particular groups of 
factors. In. a static situation, this type of division 
of labour may present no particular problem. But the 
pr-esent need is for training to play a part in the mana- 
gement of change, and to' do this effectively the diffe- 
rent levels neei to be integrated in practice. Without 
'an effort in this direction there will continue to be 
training programmes directed at immediate remedies for 
immediate' symptoms. Training policy for agricultural 
workers leaving the land, for example, has been to pro- 
vide instruction in industrial operator jsl^ills, super- 
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vision and managemerjt practice. But if these training 
programmes run concurrently and over a limited period, 
they fail to meet the .real needs of change. Such an 
approach may be characterise'd as a training policy 
si)lution. By focussing on draining. a graupt^ of people 
for a group of available jobs, ^it, fails . to .allow for thg 

Vfa'ct that people and jobs are not static. A "manpower 
\p6licy solution" would be to integrate the various levels 
training over the appropriate time-span*. From the 
point of'viev of industrial 'systems .this allows for the 
needs of manpower succession and from the' ?ndlvidual * s 
poXiit o-f view it recognises that he may. have a career 
rather than a series of job6. ^ 
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CHAPTER 2 



EVALUATION OF THE"" TRAINING PROGRAMME 
* [ (a) An overview . 

A - Evaluation in 'the Socio-Economic Domain 

. In this broadest 'Of domains, what should be sought 
is most open to dispute, and .what may be achieved most 
open to, doubt'. The successive interactions of training 
outcomes witft other fact^ors in the "various dbmains de- 
feat any attempt •tl..eValuate those outcomes in a strictly 
conj^^olled f ashi(5rtJ^At::. the same time, the evaluation of 
theends of tralulAi^^itetriate directly to ,ihe sphere oT' 
political decision. '1 ^ ^ 

A decision to implemei^ a . certain 'training project 
may for example, entail de'ci\ions about where people live 
^nd hence -about housing polic es or t ransport investment, 
/Less tangible but just a'S real are, costs and penalties 
arising perhaps rrom the splijf^ting up of families by age- 
group, urban, congestion and liCe-style etc. Are the fami- 
lies of inner-London inherently different f^om ji5^her 
people ? Yet teachers are paid a premium to -work: in 
their 'children'* s schdol,s._ ... 

, To take a more directly industrial example, if 
'workers fight redundancy by taking over their factory, 
is it worth investing government money in- training pro- 
grarpmes for' them ? Apart from the immediate' financial 
< conST3^ations of expected trading profit, savings, on 
social security payments etc., what value, is to be 
attached to the. new forms of industrial life which such, 
events may lead to ? Or .to the 'Sifference in moraj-e . 
between men who have saved their own jobs and men who 
have been left unemployed.. ? 

Evaluat3:on of human outcomes faces always the -pro- 
blem of putting on a common scale the concrete and the 
intangible or, at best, the vague and the definite. 
Political vision, exercised thrcJugh -the ndrmal processes 
cannot be replaced by any technique which social science 
has to offer, but the evaluation oC. training may be the 
means of adding to political j-udgement * the techniques of 
. social investigation*. Its most ^importarlt effectNmo^t" 
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be not merely to ^^xtend the use. of social surveys and 
sociological analyses by governments, but to introd-uce 
the idea of ^planne'd social experiments. Until now such" 
"experiments have -scarcely merlt^ the na'tn^, being 
precipitated by interestg which pre-empt the results. 
Consequently, there is generally little serious attempt *V 
at evaluation and it is 'left to historians to present , 
their results to .a later generation. Neither are small 
pi lot schemes enough in themselves . Paradoxically , phe 
restricted project designed to eliminate some of the ^ 
variables may be more difficult to interpret than a 
large-scale^^pro ject ., In the social field, as in others, ^ 
ef^f ects' do /not always "scale-up". " ^ 

The task in the so^cial Evaluation of Vocational 
Training therefore is to pu'rsue^the conseq^adnces'^^f . 
straining across the normal divisions between policy 
areas and to bring such diverse instruments as Cost 
Benefit Analysis,' attitude inventories etc. to bear on 
a'unified investigation of social events. Only by broa- 
dening the -scope of evaluation procedures can decisions 
reflect the nature of the realities they are intended . 
to deal with.' The resultants of training actions, can 
be analysed into economic, socialuandf human cotfiponents . 
Having achieved • -these ^distineti.ons^in principle , the 
implementation of appropriate evaluation prograjnmes 
should not be allowed to founded on the objection that 
we lack the means to acquire the appropriate data, f 
Social inveptigat-i*on will never stand on the same l4vel 
as a natural science 3*f^waitlng "for more "scientific" 
instruments is \theref ore pointless , The determination 
to broaden the scope of evaluation i^ the primary need. 
The techni'^ues and' instruments current^ly^ available^ will 
supply -enough date to stimulate 'the political imagina- 
tion. . ' ' ' 

The paper by Raven on "Humax/Resources, Theft ' 
As^sessment and Devwaopment" raises wide issues as tcT.the 
kind of characteristics which shohld be developed ' in . ^ 
trainees for our complex inte'rdependent societies. It 
discusses the . difficult ies experienced by the educatio- 
nal system, the motivation of trainees and the .social 
selection and placement function of qualifications » 

The case-study from Luxerabourg as ^ good example* 
of how a tr^aining system can cjpcome distorted from its 
origihal goals. The study describes how it became 
necessary to 'introduce a new Certificate in Practical 
Skill in order to implement "a method of developing 
practical skills, manual intelligence and skills for 
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certain prpre.'.::jion3 which requir^^ no fancy^ abstract 
theoretica.1 knowledge". It ^was found that* the existing 
Certificate in Vocational Skills was too difficult for 
a substantial (up to 50 number of' trainees, de^ite 
a growing dema,nd for skilled works. 

'-gjj^nally a note from Denmark, contributed to ^^he 
seminar by C. Jorgensen argues that . training wiir'in 
the future have to pay more attention to educational 
needs, which he enumerates as 

■ (i) Vocational/technical - the need to learrl^tOx use 
new technology ) 

(ii) The motivatioTial^problem, lead^g to the adop- 
tion of job-enr jrchment and job-enlargement- 
schemes , ^ 

^ (iii) Tl\e ^cial aspect - the ability to adapt to 
change ; tio new educational standards. 

B - Evaluation in the manpower policy domain 

Since the time lag for feedback data is relatively 
great in this domain, post-hoc evaluation by itself is 
unsatisf actc^ry The primary need i<s therefore -^f or 



foreca'sting -techniques buiJ.t up on previously acquip^i;; 
data; At ^p^j^esent, 'She 'neces-sary data 4s not avai laib)^^^^^^^^^ 



have turned out to be far from accurate. ( c . f . teache1^^->;:f^/ 
in -U.K.). A- significant factor contributing to this :'^,4'<' 
unreliability is technological and social' change , for 
■instance the oil shortage -and consequent undersea dril^:;^/-- 
ling, or the increase in air-travel durj^n-g' the i960' ^a/.. 
Some of the social changes presently affecting the ma€- 
powei' system however are known and their, future effects 
can be ant^ic ipated . - ' 

There is little derailed knowledge ^f the job=».vJ 
patterns and career rstru^^ures that are f'ollowed ^by /^ihst 
working people. So far, classification has "stopped a£ 
lists of ' j ob-titles and descriptions - another exampre 
■ ol^a stric^tiy "training policy" approach. 

The |:)aper by. G.' Ducra^/- outlines the growing diffi- 
culty of matching training to employment in a modern 
society. It reviews the various studies , particularly 
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n carried out 
ray stresses " 



/ ^ . - 

of a macfo-cconornlo nature, which nave beer 
in France since 1971 on this problem. Duci 
the flexibility caused by, the role of the /enterprise in 
choosing what technology to adapt to production, thereby 
■ generating specific manpowe^* (and training) needs. 

> - ' J 

•■'^t present the concept of brpadr^sed training is 

frequi>ntly invoiced where predictive power is lacking. 
While \broad- based training has its own particular vir- 
.. tues iru terms of the quality ofCthe individual's lear- 
"^^nihg, 4\ does not offer a universal safety net. It Is 
satisfactory neith^^ from ^he point of view of the 
employra!bnt system nor. from the point of view of the 
individual learner to attempt training whicfi has vaguely 
defined outcomes. Adult trainees 'in particular respond 
sharply to the de-gree of perceived relevance in a trai- 
ning programme. ^ 

Broadlyt based tjraining and predictive evaluation 
in the empim^ment system are essentially complementary 
.ther^efore. ^nly by Increasing cSur understating of the 
latour market can we define tj:"ie Tfecessary base on'waich . 
to oroaden training for change. . . / 



C Evaluation within the training poilicy domain 

» Evaluation within the training policy system has 
implications in two directions. It is a sub-ordin^te 
element of the evaluation of manpower system outaoin^s' 
and it is the *super-ordinate context for the eVa/luation 
of t refining programme results . t ' -^jj 

Studies in thl^/dCmain take as their focu^^' indivrf. 
du:^ls in the jobs for which. t:;ey were traii:Ted. 'K^e data^ 
collected usually falls into one or more of three cate- 
gories : < ' ■ 

1 - Measures of- job competence ; e.g. production 

records or suoervisors ^ ratings, promotion. 

a V ^ 

2 - Measures of \1ob satisf aPcion or morale" ; e.g. 
' qijestionnaire^, lateness a^nd absenteeism records. 



3 - Measures of employment stability ; e.g. sur- 
vival" rate of a groub of trainees,^ exit 
interviews. 

Categpr'y (l) data is aimed primarily at vali*dating 
the conteM and method of training, while category (2 ) 
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and (5) ciata'^art) (Concerned more with the selection of 
trainees for that type of training. 
« * . 

The difficulty of drawing the appropriate inferen- 
ces from labour turnover data has already been referred 
to. In fact' similiar problems arise in the interpre- 
tation of eat>egory( (1) data. Job performance is one of 
the most potentially misleading sources. of training 
validation.' It has been found for instance that young 
female employees who were thought to be insufficiently 
r skilled after training were in fact simply ignoring the 
' incentive scheme : they were obliged -to surrender their 
wage-packets at home whatever their earnings. Group 
production norms are also a frequent determinant of an 
individual's performance, especially in the case of a 
newcomer from the training school. Thus, whether em- 
ployment data, is referred upwards to the s,up*er-ordinate 
category (manbower sysetem) or? downwards to the sub- 
ordinate category (training programmes) &imijiar >dif f i- 
culties of interpretation arise. 

Th,e problem of employment data in the contei 
the manpower system has been discussed in the pi/evious 
sect&n. Data fj^m job-performance can only be inter- 
preted in the light of the total influences which^ct 
on an individual performing his job. These inCluO^the 
climate of ^ the ^employing organisation, norms and 
expectations oft.ai^ immediate work group, incentive 
sche'mes, quality of supervision and the personal nature 
of the trainee himself. The nature of many of these 
factors imposes the need for personal interpretation 
the evaluator, but certain formal meaptrres can be takei^ 
in 'order to indicate what factors mayV.be significant' in 
any given situation : \^ 

1. Measures-of performance obtained under "training 
school" conditions' can, be compared with production 
performance. * Large and permanent falls in perfor- 
mance, particularly among previously high performers 
indicate the need for an investigatipn of working 
conditions, pay structures etc. * * . 

2. Further evidence may be obtained from comparison 
groups. If persons with similiar . training take jobs 
in different production units or in different firms 
there may well be differences between job performance 
data from these grojLps. Indeed it is normally to be 
expected that there will be such differences and that 

* they will ol^ten be su*7S^antial . It is this- which" has 
led training'^ese&rchers to regard, on-the-job p|^for- 
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mance ui; a deceptive criterion of traii;gjK^ Validity 
unless it- is well ' (Controlled and caref u5ly 'inte^r?- 
preted . ^ P ^"""^ 

^ • The need for interpretation referred- to here arises 
from the practical limitations of the purely experimental 
paradigm. The rationale of experimentation depends upon 
the "control" of variables. That is to say comparative 
data- should be available to indicate unambiguously what " 
effects are associated with what circumstances. All 
that is then left to interpretation is the conclusion 
that the association is causal, i.e. that . the circum- 
stances cause the events. /rt_ is however, seldom possible 
to, -achieve a hi'gh level, of such control in human affairs. 
Thus rjob per'f (/rmance s^Mies arevnot based on groups of 
individuals wll9aj^^---rd^ntical^ except in one or two 
specified ways : they a/re basen on groups of people whose 
nature and circumstance^s (of wo;/k, of training, of 
education etc.) differ iK many ways. The need to inter- 
pret data is there^iwi^ incsreaseii so as to account for 
vihat cannot be control led . 

The study by P. Waldman sKov/s how training aimed ' 
at "versatility", produces very dlsCferent effects ih'-two 
firms and underlirnes the essential re la tivisflf-'-evf trai- 
ning evaluation. In different circum'stances difT^ent* 
relationships obtain betweenVsm^erf icially similiar\ 
variables. In this parti culaV^udy the important 
circumstance' is the climate o'f\^he firm : it coulds, 
equally be the production system"*"^ the technology etc. 
In any case' it is vital that the . circumstances from 
which data arise be taken fully into account since it is 
only the light of such an account that the relationship 
betwee-n variables (e.g. training method and Job plrfor- 
mance) becomes clear.. 




(b) HUMAN RESOURCES : THEIR ASSESSMENT ANP' DEVELOPMENT 



1 - The Sco^e of the Pap#r and the Impor^tance of the 
PFo^fem ^ ^ 

■ ~ ■ ' \ 

I have/ been asked provide a conceptual t^^^per 
on the shor\ term' evaluation of vocational tr-aining. 
In order to Ido this adequately, it is necessary to set 
the discussion in a wider context, and, in particular, 
to set what I have^ to say in l:he context of a discussion 
of such questions as ": "V/hat are thex^oals of the 
educational programmes which are tcy be evaluated ?" 
"VJhat are-', the students expected to/ be able to do b€;fct^er 
as .a result of the courses ; V/hatycompetences are to be 
fostered ?" "What is the" psy c^ioibgical nature of these 
competences ?" "How can thei-p-^resence or absence be 
detected ?"'and "How is one*, to find, out what the long 
term consequences of people haying developed, or failed 
to develop, these competencea may be ?" 

Althougri the paper will be broa^d in this sense, it 
/will be very limited in two other .senses . Firstly-, it 
relates to summative evaluation designe'd (a)- to provide 
appropriate certificates for students and (b) to develop 
a better understanding of the nature of the goals to be 
achieved, the way they are to be achieved, and the conse- 
quences which different types fcf programme are likely 
to have. This inf orfna-tion is fexpected to be of value in 
'improving future educational programmes. - It is not - 
^intended to improve the particular educational programme 
^in. the courge of which it is conducted. The paper is 
not c^cerned with the many very useful types of formative 
evaluation wi^iich could be carried out within a particular 
programme in order to -^^iR^rove -it . 

Secondly, the paper is limited in the sense that the 
'main theme of 'the paper relates directly only to tHe 
evaluation of one sub-sector of training programmes, a 
sub-sector vjhieh, at preseg^, forms 'only a minority of 
training programmes. This sub-sector is made up of 
programmes intended to foster such qualities as versati- 
lity ,' confidence in one's ability to .master new tasks, 
the ability to learn without instr,uction (a very diffe- 
rent thing from learning to take further formal courses) 
and the' ability, to build up a picture of an overall pro- 
gramme activity, and one's part in the whole, without 
having .to, be given detailed instructions. 



In emphaGising that the-jjaper relates directly to,, 
only tone sub-sector of vocational training ;it is not 
meant^ to imply that the ■ paper ' sS^relevance td vocationa-1^ 
training as' a whole is slight. Exactly the opposite i-6. 
the. case. There are twc/ reasons for saving this. The 9 
first Is that eummat ive e valuat3.ons of th% sort which . 
will be discud.sed are essential [if educationalists as a 
whole are to'' break out of the treadmill' situation in ; 
which they iUnd themselves. More will -be said' a/^out 
this in a moVient. Th,e. second is that, although the pro- 
gramme to vmich the paper relates directly form^N.mino- 
rity of programmes at. present, they are likely td^form 
the majority in the -future. More" and more, our cbmplsx, 
interdependent, ' societies -will ^de*and ^ th,at peoplelthink 
widely and responsibly, take per:sonal ' responslbi li^tj^ 
for their decisions rather than refer to authorityF'take 
the initiative in sta;:"'ting chains of development, aod 
monitor the effects of- the action so initiated^^ order 
to take\corrective- action when necessary. E^en today if 
one w-antkan efficient bus service oi^e requires- bus ^ ^ 
driwns wnb are able to think l^i a wide and responsible 
f^^hion aM able to relate effectively to a wide range • 
ST different types 'of people performing different rolee 
n.n society . (Van'Beioum, 1968).. And one needs- managers 
and Cor'emen -who habitually consider the social impLica- . 
tions of their decisions. 

Let me now amplify the first reason I. gave' for 
focussing on this topic. Let me briefly try to justify " 
the staterraent that more attention will have^to be -paid 
to summatrvei evaluation in this area if all connected 
with e^ucatiiDn are to be able to get out of the highly 
'unsatisfactory situation in which they find themselves. 
At -the- present time some teachers find themselves . tea- 
ching toward goals they do not believe to b'e very impor- 
tant and neglecting the goals they believe to be mqre 
important .(Raven, 197^)- 

In the next section. of the paper, j will explore the ^ 
reasons/for this in more detati.i-, and will suggest that 
the same thing' is likely to' hkppen to those concerned 
with vocational • training unlessN^ore atterition is paid 
to evaluation in the areas which "h^^^e^^be^tt^entioned . 

What !• am suggesting is that-.the central\ and most- 
critical problem in evaluatioh - in the evaluation of 
vocational trai: and in the evaluation of education in 
general - is the prcblem of specifying the nat/ure of, and 
developing the.me.'Uis with which to assess, the competen- 
cies which it is most important that school^ and vocatio- 
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"rial tr'airiir:^:; proir^i'-iuuir .:. .^houlo be- concerned with foste- 
ring. ' • 

. . ' ' - t ^ y 

In the course the paper it will be argued that 
it has been the failure of educa" t iona lists to tackle , 
this problom in the p.'^t which hag led to everyone con- 
nected with e'^cAtion - and not. just teachers and 
pupils - being disti^actea from the goals they consider 
most 'important and to their focussing on goals which are 
less important. -. ■ . ' - 

In saying that not only teachers and pupils have %. 
been af^ected by failure to tackle , this problem 'mv inten-^ 
tion is to draw attention to the fact :that vast areas of 
social Dolicy have heerl , af f ected^as w.ell*: grossly mis- 
guided attempts have been made- to counter poverty, -^^social 
disadvantage, 'and unemployment, precisely because educa^- 
tionalists have failed to consider the ii-^ goals, the way 
iVi which they are. to be achieved^, the way their achieve-* 
meht 'is to be as^sessed, and' the consequences o^' the 
competences • devel-oped" for 'the individua'^ls ^concerned arjd 
the society irj ".which they live. . " - ' 

If this situatio.ri*;is to be rectified, it^ .seems to rfe 
tha\ we must ^.develop ways ;or evaluating progress towards 
the goals of i^^ycati^sp'-^which realiv are the most impor- . 
tant. goals; And-'p^^^ems to' me - and I have ji^trieved a 
certain amount of' evidence tc^ back- up my position from 
literature - that the -teachers, pgipils, ex pupils and 
parents thaf ^have interviewed are quite right v;hen they 
say that these goals include fostering such qualities as ^ 
'V^he abM-ity to work with o.t'hers, initiative, and concern 
to work efi'^ective ly for the^-'gooci of the community in 
wMch we r^vu. ^ 

/ For reasons whi^h I will discasS^/iri; a ■•^fffdlTierft, it 
seems to me vital that,, if the siti3&tion 'l have described 

,is to be rectified, and if the same distortion is not- to 

'occur in the section of education concerned with voK^ati'o- 
nai training, we must, in thinking abc^t evaluation, and 

■ in carrying out evali^tion exercises, resist the tempta- 
tion to retreat into measuring progress toward goals ^ 
which, while easier to measure, are less important.^ 

We must resist this temptation because, in spite of 
the apparently innocuous nature o'f a decision to foliow 
this pkth, the decision is," in T^ct, ^'^^i^eciaion -of im- 
mense ^ci,ological significance. It b^^gs ih its train, 
as surely as nigftjb follows day, a whGl^''^6-^t>VQf important 
social consequences. 
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.Failiiro to resist-, the temptation will result in 
educators, occupational selectors, staff developers, 
organisational developers , and policy makers who -^dea 1 
with issues ranging ffom regional development to^ivil 
rights, focussing 'QH the goals of education," and on the 
human resource qualit ies which are assessed, and gearing 
their policies and thinking to t hose qualities, and 
those qualities alone. As a result they will necessfirily 
neglect to focus on , foster", and consider , the characte- 
ristics v/hich.are really the most important from the 
point of viev/' o f ' Irmjroying society as a whole, the orga- 
nisations cone e,r ned , and the 1 i ves of the individuals ■ 
directly involved. 

2."^ - The Social Functions of Qualifications 

What jLve my reasons for arriving at the 'c^^lusio;i 
I have stated ? The ■ processes just described ^rise 

from th-e* soc ial , not the educational, funct ions 'performed 
by educational qualifications. Teachers and pupils are 
in f act ^e 1 1 \ aware of the social selection and placement 
functions of qualifications, but thev have'^rarely consi- 
dered explicitly the implications of 'this function for 
their own behav-iour. Even less frequently considered' 
are the implications for the interpretation one 'places . 
on the mushrooming demand for education >which we have 
witnessed over the last quarter df a ce'ntury . 

» 

Educational qual if i-cation s perform the function of 
allocating people's life chances, and legitimising the 
way in which the flow of individuals into privileged 
positions in society is regulated, regaJ^rdless of the 
value of the actual content of the programme, on which 
this certificat-ion is based. other words this func- 

tion is performed without ajny regard at all to the -value 
of what has been learned in educatio^nal programmes. As 
a result, the level .of demand for qualifications - 
whether that demand is expresse^d 6y students or by pros-, 
pective employers- - cannot be treated as evidence of the 
need for , educational programmes. It may 'be evidence of 
the need for a-qlegitimated allocator of 'life chances, 
evidence of the need for changes in the institutional 
structure of society, evidence of the heed for. economic 
growth, or .evidence of the need for a re-distribution of 
wealth. But that is another matter. In and of itself 
such demand cannot legitimately be construed as evidence 
of the need for the skills and competencies foster d in 
the progi^ammes for which there' is a demand. 
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It is th 1 1:, L^.e loc t ion and placement f unct ion of 
educational qualifications which has led to the over- 
•whelming growth in demand for education which we have 
witness-ed for the last quarter of a century. This demand 
for 'qualifications has risen unabated by the fact that 
the competencies fostered in those programmes were not 
the competencies which were, in the opinion of the tea- 
chers, pupils^ ex-pupils, and parents involved, the ones^ 
which it was most important to foster, and in spite of 
the fact that there was' scant evidence that they would 
lead the individuals concerned to be able to perform 
their jobs more effectively or to lead their lives with 
a greater sense of fulfillment and 'sat isf act ion . (Berg, 
1971, Jencks, , 1973 , Raven, 1973). . 

The fact that certification -; and "therefore the 
selection and placement functions^ of educational qualifi- 
cat iojfS - was based on the at tainment of the goals which 
were e^asiefet to assess, although the importance of these 
^als was widely .-quest ioned , meant that teachers and^ 
students had to focus on achieving the goals which were 
.not assec^sed but which were most important. Had these 
teachers and students' spent their time, working toward 
the mo3*e important goals, the students' life chances, 
and the teachers^ reputation, would have been jeapor- 
d ised . 

/ • ' ' * ' 

Confronted with ever increasing numbers of 
"qualified" applicants, employers reacted'by raising 
their entry "requirements" -. still specified in the same 
t.erms - ther-'^ y adding fuel to the fire. 

Employers, like teachers (who also knew that there 
v;as something very wrong witl^ what was happening in the 
educational system, even if they could not quite put 
their finger on" it) continued to play their part in this 
conspiracy with some degree of soul searching. Although 
they often knew in their hearts that the criteria on 
which they based th^^ir selection procedures were not 
those which were really the most important from the point 
of viev; o'f performance, reliance on the criteria they 
used at least enabled them to avoid the accusation that 
they were favouring the sons of friends, or those who ^• 
had "respectable" backgrounds, over and above 'those who 
came from less prostigeful backgrounds. 

If, therefore, we are to make much progress in 
assessing the need for training programmes^, we would do 
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well to treat' otuti'ctics relating to student demand^^f or 
courses, or- which deal with employerc "requirements", 
v; ' ruspiciori . 

More thaTNthat. In any attempt to evaluate the long 
ter'm efrects or* training programmes we must attempt to 
separate out the consequences wbich arise frojn the skills^ < 
developed in the cour^^ of those programmes from the - 
consequences of certification and placement itself. If' 
this is to be done - and this is the point I have been 
leading up to we must be able to assess the real 
human resource competencies - or incompetencies - foste- 
red in the course of educational programmes in the short 
term. We will return to a discussion of the nature of 
these competencies and' the way in which they are to be 
assessed in a moment. But first, I would like to- have 
a brief look at one more issue. 

3 o- The coni.oept of Continuous Education 

■X. <• 

If the arguments previously put forward ar-e accep- 
ted, it is necessary to seriously re-examine the thinking 
which has penetrated official f^orts on *the future, of 
education such as the Fauwre report. The, 'admittedly 
imperfect, -evi icnce collated by , the authors mentioned in 
the last -^ction strongly suggests that formal educa- 
tional Gualif icacions correlate neither with job, perfor- 
mance nor with performance in further formal courses. It 
is therefor-e important to seriously question whether what 
is needed in the course of life-long education is further 
formal courses of the traditional sort or whether some- 
thing very different indeed is required. The sort of 
changes which may be needed can he ' illustrated by taking 
a*few examples. Teachers generally assent to the propo- 
sition that they wish to help. their students- to "learn 
to learn". However, further probing reveals that by thio 
they normally mean that they wish to 'help their students 
, to learn how to take further formal courses.' As an aside 
it may be interpreted that Berg^s work suggests that they 
are not even successful in achieving this very limited 
objective. In contrast^ what they might mean is that, 
they would like to teach their students how to make their 
own observations""; to learn how to learn without ins * 
truction . This might be an altogether more useful quality 
for pupils to develop.^ 

If thi-s were what the phrasa^ "help students to learn 
how to learn" meant a number of Questions would arise : 
(a) through what mechanisms cio l^achers at present seek 
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to hel'p their students to develop this ability ? 
(b) where is the evidence that their students do develop 
this quality ? (c) how might the presence of this abi- 
lity be more effectively assessed ? (d) through what 
sort of programme might is be better. fostered ? 

4 - What Qualities might. Students need to'^develop in the 
Course of Training Programmes ? 

The discussion in the last paragraph may be taken to 
imply that one of the qualities students may need to 
develop in a course of in-service training might be "the 
ability and the willingness to make one's own observa- " 
tions and learn without instruction". But how do we 
know whether this is^ an important quality for partici- 
pants to develop ? An impressionistic analysis of the 
qualities required to perforn? well in the job^ entered 
by school leavers, supported by data collected from pu- 
pils who left school five years earlier, will be pre- 
sented. This -analysis and data supports the view, shared 
l5y the vast majority of teachers and pupils, that the 
main goals of education should be to foster such quali- 
ties as initiative, responsibility, decision taking . 
ability, judgement and *■ forecasting and planning skills. 
Attention will also be^ dr^wn to the evidence suggesting 
that ^B?T^e qualities areSoBsential to the effectiv-e 
performance -of jobs as distinct as driving a bus (Van 
Beinum), by medical practitiorrers (Taylor), by managers 
(Egan), and by researchers (Tay-lor). Finally attention 
will be drawn to the fact that assistance in developing 
these competencies is particularly- required by indivi- 
duals undergoing promotion - (eg from researcher to - 
research manager) and not only by individuals who find 
themselves out of work for one reason or another.. 

This discussion leads to three conclusions : 

a) That the qualities to be developed through educational 
structures, both child and adult, are more in the 
realm .of developing qualities like the ability to work 
with others, the ability to understand social institu- 
tions' and-get them to work effectively, and the wil- 
lingness and the ability to embark on a necessary 
course of action without knowing 511 the steps to the 
goal in advance, but equipped' with^ the ability to, 
monitor what happe'^ns, learn from it, "and take co^rrec- 
tive action when necessary, than it is to develop the 
ability to manipulate mathematical equations, or a 
brake, clutch and accelerator. 
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b) The number of people who need to be- provided with 
opportunities toodevelop such qualities greatly 
exceeds the numbir of "unemployed". 

c) The structures requiz^ed if people are to develop these 
abilities differ greatly from the educational struc- 
tures to which we have grown accustomed in the past, :y 

5 -^ What is the Nature of these Qu'ajities ? ' ^ 

If courses directed toward- developing these .quali- 
ties are to be provided and assessed we need to know a . 
great deal more about their nature. After all, teachers 
at present consider that the development of these quali- 
ties is the main goal o.f education, but they say thatj 
in practice, they do little to foster them and that out- 
comes in^this area ar^e pitiful. /i,^ 

In the main paper it will be 'argued that these' 
qualities are best thought of, not, as abilities (such as 
intelligence or mathematical abilitf) but as mot^i.va tional 
dispositions . In other 'words, the defining chara^jteris- 
tics of these qualities is^ that the individual engages 
in them spontaneously ; they represent the willingness to 
use such abilities as one' -has to achieve a valued- goal . 
If we are. to understand their nature lor- index them) we 
therefore * need to understand two components of human 
behaviour which have received scant attention from psy- 
chologists or educationalists in the pas,t. They invol^ve 
understanding values and understanding qualities which 
enable an individual to organise his life to achieve his 
valued goals effectively. -A detailed discussion of 
valued goals and .pf ■ the components of competence (which 
have been hinted at, above) will be presented. 

•\ 

6 - How can these competenci.es be fostered ? 

It follows from what has been said in the last para- 
graph that fostering these qualities involves providing 
people with opportunities to clarify their values and 
with opportunities to practice and develop new styles of 
behaviour which- will enable them to achieve their valued 
goals. The 'former may best be >done by presenting them 
with research data (most of .wl:^ich. we. da not have.), while, 
the latter may best be done through such things ,, as, role 
playing exercises. 
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7 - How c a n . thu :l; o' q ua 1 i t i e s be . a s s e s s ed ? 

If it is true that these qualities are best thought 
of as motivational dispositions ^iti^follows that the 
techniques of assessment which are appropriate are those 
appropriate to indexing motivational dispositions. In 
. other vJords , it is necessal^y to find out whether people 
show a spontaneous tendency to engage in the components 
of effective behaviour. 'This can only be done if we 
know which goals they value, for they would not be ex- ^ 
pected to engage in these complex a/ctivities in relation 
to/goa.ls- they did not value, Teacmers^ failure to ask 
whethe-r students value the goal toward which they are 
expected to work may'account for the poor predictive 
validity of the assessments teachers make once one gets 
beyond the next stage .of -education. In othj^r words a 
pupil may display fev^ of these characteristics in rela- 
tion to performance at mathematics (and hence do badjy 
at it) v/hile being able to display many of them in rela- 
tion to a goal/he does' value (such as football). 
Furthermore if he later comes to value performance at 
mathemat^ics because tie needs- it to achieve a goal he 
values he may come to do very well at selected branches 
of ft(Berg 1971). Stevens (196O), Hernstein.^ ( I97I ) and 
the author have shown that the majority of secondary 
school pupils see little point in their school tasks 
other than as a route to gaining certificates which will 
act as passports to high status occupations .» When one 
also understands that many of them do not v^lue entry 
tp.^these high status occupations, the lack of "motiva- - .6' 
tion" so "Of ten commented upon among 'secondary school ; 
pupils oomes as no surprise. ^ 

A particularly fruitful way of establishing whether 
people are willing to engage in the patterns of thinking, ' 
feeling and behaviour ^ connotated by such terms as ini- 
tiative, independence, and responsibility may be to ex- 
plore the CHDnseqaences they anticipate if they were to 
engage in these activities . (Raven 1975)- It may be pos- 
sible' to e licit and assess these anticipated consequen- 
'ces more • successful ly fc^Llowing the methodology developed 
by Fishbein (1968). Such assessments should reveal : 

a) The strength of an individual's v/illingness to engage 
in these behaviours in pa*rti cular situations . 

b) Problems which woulc^need to be tackled in in-service 
courses. An indi-^idaal may need research data to 
demonstrate that certain consequences do not follow 

32 > 



from engaging in a particular type of activity ; 
alternatively he may need assistance in developing 
styles of behaviour which ensure that the currently 
correctly anticipated conseguences no. longer follow, 

c) Changes which may need to be made in the organisatio- 
nal climate of the employing organisation if ,such 
behaviour is to be released : If an individual belie- 
ves that the consequences of displaying initiative in 
his firm will be his dismissal (and many people do 
believe this) then it would behove the firm to re- 
examine its promotion procedures . A recent study , 
showed that the army were in fdct promoting people 
whcr actually possessed most of the qualities they 
explicitly recognised they did not wish to favour, 

8 - How c&n the Consequences of these Qualities be 

assessed?- / 
" 

The consequences of having developed or not deve- 
loped "these qualities deserves to be studied at three 
levels : (a) for the pattern of life accomplishment, 
way of life and pattern of life satisfaction and frustra 
tion of the individual concerned, (b) for the organisa- 
tion in which he worjcs and (c) for the society in which 
1^ lives. 

Different me thodologies ^ are appropriate at these 
three levels. However, perhaps' more important, the 
consequences will differ depending on the institutional 
structure of the organisation and society in which the 
individual lives and., works. Attention will be drawn to 
a number of organisational and community climate varia- 
bles which moderate the effects of these motivational 
dispositions. 

9 - What Alternative' ways of Fostering these Qualities 

might there be ? 

As was Indicated in the last paragraph, the 
institutional structure in which an individual lives and 
works has the potential to release or frustrate these 
qualities. It also has the potential to foster them. 
We earlier spoke of the need for training programmes to 
concentrate on fostering such qualities rather than on • 
specific , occupational skills. We also draw attention to 
the fact that an enormous number of people could be 
expected to benefit from, such programmes. The paper 
will conclude by remarking that some ways of organising 
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firms, some ways of organising regional developmeiVt 
programmes, and some ways of organising^ community aevelop 
ment programmes, would be expected to" have a^much more 
beneficial effect on the development o^ theae' qualities 
than other ways of organising the-se things. All will 
have an effect'. The question is. which ways of organi- 
sing these things have what effects on the human resource 
competencies of the population - on their perception of 
the institutional structure in which they l^ve and work, 
on their expectations of their own role in relation to 
that, structure, on their expectation «is to the role of 
others, and on their willingness to take on themselves 
personal responsibility for engaging in patterns of 
behaviour which are socially important. In the end the 
Social Fund might best be spent, not on running courses, 
but on enabling organisational and community policies to 
be carried out in such a way that people learn "on the 

job'y 

10 - Summary : The Evaluation of Vocational Training 

Programmes, Staff Development Programmes 
and Organisational Development Programmes 

The implications of what has been said in the course 
of this paper for the evaluation of vocational training, 
staff development, and organisational development pro- 
grammes may now be summarised. 

(I ) There is an urgent need to focus more explicitly on 
the sort of qualities with which we have been pri- 
marily concerned, that is to say on human resource 
competencies, on motivational dispositions. 

♦ 

(II) Since these qualities involve a values component, 
any attempt' tb foster ,them more effectively must be 
based on better research data on the personal and 
socj^l consequences of people possessing, or not 
pos^ssing, these characteristics when these people 
live and work in different institutional structures. 

(III) If we are to develop effective ways of assessing 
;these qualities we must pay attention to the fact 

that it is their self -initiated, spontaneous, nature 
whi ch it is most important to assess. The critical 
feature of these qualities is not so much whether 
the individual can engage in t-herrf^ as whether he 
tends to do spontaneously. We must think of them as 
motivational dispositions... The methods tb be used 
to assess them are .therefore the methods appropriate 
to assessing motivational dispositions. There is 
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theif'cl'or'o a great need for a substantial increase 
in investment in research and development in this 
area. 
> 

(IV) Means of assessing these qualities are essential, 
notV4nly in order to evaluate vocational training 
^)rogrammes, -general education programmes', and 
staff and organisational development programmes. 

V They are eesential pre-requisites if an^^one whose 
work Impinges on, or involves, these qusSl^ties, is 
to be.^able to pay attention to them.' Iry other 
words, it is .elssentral to develop means of asses- 
sing these qualities if teachers, pupils, instruc- 
tors, rstudents, personnel managers or organisational 
development consultants are to get recognition for 
their';efforts in the direction of fostering these 
qualities, encouraging their staff to display them, 
• developing regional policies which will encourage 
them, or taking their existence and development 
intt> ac^30unt in social work- W ithout- measures of 
these qualities all these groups of people will pay 
attention to the qualities which are assessed, less 
important though they may really be. It will be 
^ only for fostering these that they will get reco- 
gnition, ,and these less important qualities alone 
will figure in the research studies on which policy 
is based and in the discussions of hard-nosed poli- 
cy makers who are only prepared to talk about real 
(ie detectable) qualities, rather. than the numbo- 
jumt)o of trick cyclists. 

(V) IX is not even" possible to assess the long term- 
effects of vocational training, general education, 
or staff or organisational development programmes 
directed toward these TOals . without measures of 
these qualities. Without" such measures,, the effects 
of training programmes will be inextricably bound 
up^with the effects of the social s^fel^c tiorfi^ and 
placement inunction performed by educational creden- 
tials. To carry out an effective long term evalua- 
tion one must first be able to detect short t^m 
differences. 

(VI) The long term consequer^ces of people possessing, or 
not possessing, these qualities will vary markedly 
with the nature of the institutional structure. in 
which the individuals live and work. The evaluation 
of training programmes directed toward fostering 
these qualities therefore implies th"e~Tieve lopment 
and use of better indices of the nature of the ^ 
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instLtutionai structures in which individuals live 
and^work. ' 

(VII)The most effective way of fostering these qualities 
may in fact be through the use of carefully struc- 
tured staff development, community de'^^lopment, and 
regional deve lopment programmes . To be ef f ectiv.e 
these programmes would have to be explicitly desi- 
gned to place people in situations in which they 
would develop , these qualities, «>and which would 
encourage them to engage in, relevant styles of 
behaviour. This, possibi li ty should be carefully 
,studied in any research programme which might be 
initiated . 
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(c)THE INTHODUCTION OF THE CERTIFICATE OF PRACTICAL SKILL 



A Case-study 

(a) Background 

. It has been the consistent policy of the Ministry 
of National Educatj.on, which is responsible for vocatio- 
nal training, to give our country 

(1) a modern educational system, i . e . a diversi f ied and 
graded system permitting both the best p6ssible guidance 
of trainees and the best possible general and vocational 
training according to the actual needs and the employment 
prospects of our national economy. 

A serious effort has been made to base as much voca- 
tional training as possible in the schools and to coordi- 
nate the various types of training clearly and systemati- 
cally. With the consent and cooperation of the trade 
chambers and .both sides of industry the Minister of^ 
Education has' completely and systematically ' reorganized 
the training programmes with the following aims : 

(1) to raise the general standard of professional 
".qualification and increase the technical -capacity 

of the candidates 

(2) to guarantee at the same time a humane, social and 

cultural training for our semi- skilled and skilled 
workers 

(3) to allow continuous vocational and social advance- 

ment of the candidates 

(4) to keep up with^the increase in technical knowledge 

in all fields^&hich calls for new qualifications, 
more extensive/training for aLl professional acti- 
vity and a more subtle and wider range of functions 
and levels of training. 

The role of vocation^ training appears in a com- 
pletely new light : / 

- firstly it has -to provide future craftsmen and skilled 
workers with solid basic training on several levels ; 
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- secondly tiMinees must be classified according to th^ir 
prospects hnd their ambitions ; 

- lastly vocational training must be used to the full- to 
guarantee success to young people in their chosen 
career by giving them the best possible training for 
their professipn. 

This obviously means that two requirements of a 
modern economy clash with each other at school level 

(a) growing demand for technical knowledge (qualitative 
requirements ) 

(b) demand for a ii/creasing number of skilled workers 
(quantitative /requirements) , 

The number of failures has in fact increased with 
the growing intellectual demand on trainees. One effect 
of the reorganization of technical and vocational trai- 
ning has thus been to limit the- number of trainees who 
meet these new demands. This is obviously in the inte- 
rests of neither the economy (increasing need .for 
qualified workers) nor of the trainees and their parents 
(social ambitions). 

(b) Aims .-^^ 



There can be no doubt that children vary naturally 
in their intellectual'^'and practical ability. Should a ' 
Minister of Education therefore concentrate on a pro- 
gressive system of training (which may even be called 
avant-garde in certain respects, ' an elitist^ i.e. a 
.highly intellectual system and completely, ignore those , 
who can no longer keep up with the more demanding new 
system althx)ugh they have the' necessary practical skill . 
for carrying oi^t a. given profession jDecause they la(3*^ 
the ability to assimulate the theoretical abstract ^* 
knowledge relating to it ? [ 

We feel that a Ministry responsible for the educa| 
tion of. the whole country is under an obligation to set 
up methods of training schools and institutes for deve- 
loping the quality of the population as .a whole, incluv 
ding the physically and mentally handicapped, and no-t 
merely the intellectual ' elite . 



And it would be extremely strange if there v?ere no 
room in the multitude of vocatiojjal training misthods foi 
a .method of developing practical skills, ma*nual • - 
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intelligence and .skills for certain professions which 
require no fancy abstract "theoretivCal knov/ledge , 

Should pupils who cannp^'meet the new demand be 
simply sent home, dismissed Ti^om our schools after -a 
painful struggle from failure to failure without any 
qualif icatioTi, when our economy and especially the craft 
trades are crying out for labour ? The Ministry of 
Education, in agreement with 'the 'Chamber of Trade, has 
therefore decided to divide craft apprenticeship into' 
two systems : ah overall system leading to the cert^ifi- 
cate in vocational skills (CAP) and a practical system 
leading to the certificate of practical skill (CCP) , 

(c) Training programme 

The new method of training was introduced in 1969/70. 

A. The normal method leading to the CAP passes through 
the centres of technical and vocational edu,cation and 
consists of a joint seventh year at the^age of twelve 
(after six years of primary education), an eighth year 
which serves as a observation period and a ninth year 
when the actual vocational training begins^ The follo- 
wing sections are already in operation : metal, wood- 
working, painting, hairdressing, c lothingtrade , bailding' 
trade, food industry and graphic art (printing and 
photography ) . j ■ 

At the end of this preparatory stage, trainees take 
an examination and then go on to the apprenticeship 
proper. The examination is divided into a .general 
theoretical section and a vocational section consisting 
of vocational' thepry and practical training in the pro- 
fession chosen. School marks account for a third of 
the results. , , 

cT'r'tilnees who pass both parts of the examination 
have theri? apprenticeship reduced by six months to one* 
year, depeyvling on the trade. 

Dur'ing the apprenticeship itself .their v^ational 
theory is reduced to four hours^^ week for two' years if 
they have chosen a three year apprenticeship and for 
three years in the case of a three plus two or four year 
apprenticeship. 

Trainees who pass only the vocational part of the 

examination also have their apprenticeship reduced, but 

must in addition to the four hours of vocational, theory 
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attenu rei'r'u.;iuM' ooui'.xt; in i^cneral education so that 
they can repeat thiy.p-ir't oV the examination. 

Trainees wlio paso only the general p^art oT the - 
examination can either repeat the v^'hole year or begin a 
norinal apprenticeship. Together with traineero .corning 
from continuation courGei:: they*, attend special clashes ^ 
in vocational theory (I'our hour's a,- v/eek) during the thirst 
year of tlieii' apprenticeship, , so • that' they ' can repeat the 
vocational .part of the transitional examination. If they 
succeed they will follow the normal classes but will be 
.one year 'behind their year of apprenticeship. Trainees 
who fail the whole examinatiori can either repeat t!ie 
last year or -'begi!! a converxtional :i pprent iceshi^ . In 
this case^hey, tpgether with trainees wrroxnavo come 
from the ^continuation classes- have speGia\ classes in 
vocational th.eor^y (four hours a week) and receive general 
educatipr; iri their spai'e time so that they can take the 
'.whole of the t r-ansitidnal examination' again . 

In future all cctndidates for' the CAP will have to 
have passea this transitional examination. 

Trainee apprentices in the accompanying classes 
pr^eparing for CAP (lOth, 11th and ^].2th year of part-time 
education) must pass an examination, before continuing. 
If they fail they may eitht^r^ : 

(a) repeat the year which will automat i cal ly - extend 
their apprenticeship contract by a corr-esponding 
period ; 

(b) or transfer to the training for the CCP which will 
allow them to pass on to the fallowing year. 

The examination in vocational theory at the enci of 
the appr'enticeship will cover only the syllabds for the 
last scriool year. ■ ^ ' ' 

In order' to improve coordination between school ^and . 
workshop, the examining boards will act as supervisory 
committees for each branch. In addition representatives 
of the schools will sit on the committees sat up to 
assess the results in the examination in vocational 
theory and the practical examinations at the endof the* 
a'pprent icesrii p . * ■ ■ ^ • ■ 

Apprenticeship' leading 5^0 the CAP is intended for 
t'raih'ees who- are suiT iciently intelligent to absorb and 
assimilate theoretical knowledge, a^id to allow^them to 
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continue the ir studies to obtain " higher qualifications . 
It is the normal basis for qualifying as a master and, 
under certain conditions, -as a technician or even an 
engineer. 

« 

It should be emphasized that the ninth year is the 
first year of a system of apprenticeship which gives 
successful trainees the fcllQwing advantages : 

1 - a's they begin this year of study after eight years 

of primary school, they finish their vocational 
training six months or- one year earlier than appren- 
tices who begin after they leave school ; 

2 - they obtain a solid basic training for a whole group 

of careers which will make it easier to learn one of 
this group and to change jobs later if necessary ; 

3 - the general level of education in, this and subsequent 

years is higher and therefore more profitable to the 
trainee than th(S level of the accompanying classes. 

An apprenticeship along these lines is also possible 
for trainee salesmen, storemen and decorators, 

B - Conventional apprenticeship leading to the CCP , 

Apprentices who choose this method are attracted by 
direct entry into professional life- for socio-economic 
reasons or because they are not adapted to the traditio- 
. nal school system , In principal the CCP is only an 
immediate outlet into working life. It is basically 
intended for less gifted trainees or those who do not 
wish to receive theoretical training beyond what is abso- 
lutely essential for a good workman. 

Apprentices are normally recruited among pupils 
coming from complementary education «and those in vocatio- 
nal training who have failed the examination leading to 
the CAP. In addition classes will be organized to aillow" 
gifted pupils to transfer to an apprenticeship leading 
to the CAP. 

Apprentices intending to obtalin the CCP m&^st sign 
a normal contract of apprenticeship for the chosen career. 
The accompanying courses will be orv^anized taking the 
career ir;L0 jocount in order to tea6h the trainee voca-\' 
tional theory in a concrete fashion. They will be oj'ga- 
^nized by a cup of firms, by^chambers ef trade or by 
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the Ministry of Education either as weekly courses (day- 
release system) or as periods of theoretical training 
covering several weeks a year (block-release system). jt 
The practical part of the final examination for the CCP 
will be at the .same level as that for the CAP but the 
examination in vocational theory will be combined with 
the practical part . 
r 

Special classes will be organized in order to allow 
holders of the CCP to take the theoretical examinations 
at a later date an,d thus obtain the CAP. 

This method of training meets the economy* s need 
for skilled workers and also meets the social need. It 
allovethe recruitment of a necessary and useful labour 
force and at the same time giv^s manually gifted young 
people a suitable vocational opportunity. 

Training leading to the CCP is therefore a complete 
and coherent method of training meeting all the demands 
on which our concept or vocational training are based, 
by which we mean apprenticeship and the assimilation of 
specific knowledge, regardless of their nature and level, ^ 
which a person must acquire in order to enter vocational 
life or to improve"; himse If . 

The CCP gives this specific knowledge : the practi- 
cal exa*lMnation is the same as that for the CAP, and the 
theoretijcal knowledge necessary for good work is also 
examined but not in an abstract bookish form. 

Furthermore the CCP -3 a stage towards the CAP. 
The candidate may at any glvpn moment expand his theore- 
tical studies in order to obtain the CAP. Evening 
classes for this purpose began in 197^ • 

The CCP is an innovation in the field of vocational 
qualifications. It arouse from three basic preoccupations 

of the Ministry of Education : 

(a) to meet the increased need £or highly skilled workers 
and thu^ to reorganize the C^AP and the master *s 
certificate ; 

(b) to meet the need for large numbers of good workmen ; 

(c) allow as many young people as possible to obtain a 
vocational qualification giving them appropriate 
chances in their chosen career. 
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The CCP should be' seen as the culmination of a 
normal training period in the first degree of a^real 
vocational qualification.^ 

A diploma;'.. normally f.urnishes proof of a normal 
training. The 'value of a man on the labour market. de- 
pends on hio qualifications/ 

\- 

It is therefore in the v;orker's owh interest t6 
obtain a diploma, e:;peGially one approved by the State, 
Many of the less gifted find thl^. impossible because the 
level of knowledge required is out of their reach. 

Regardless of his talent, it appears normal that 
every man looking for a vocational qualification must 
be able to obtain a diploma, provided he makes an effort. 

There must however be final or intermediary diplo- 
mas at every level of knowledge or skill. For this 
reason we divide each level into stages, each with its 
own certificate or diploma. The stages in vocational- 
qualification in Luxembourg are as follows : 

1. certificate of practical skill (CCP) 

2. (a)* certificate of vocational skill (CAP) 

(b) certificate of vocational education (BEP) equiva- 
lent to the CAP but obtained after a completely 
college-based training 

5, (a) master''s certificate ^ ^ 

(b) technician's certificate 

4. Engineer's diploma, 

( d ) - Control of results 

Before the CCP v;as introduced betweer^ 56 and of 
all trainees failed their finaL examination. 

The percentage of failures was reduced by 255^ in 
19/0, 18^ in 1971, 115^ in 1972, in 1973 and Jl^ in 

1974. ■ 

Objections ^to the CCP fall into three main groups : 

(I) 'The CCP is a, method of training on the cheap by 

which the capitalist bosses can easily exploit an 
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untraine vl labour i'orce . . 

One may reply to this that the opposite is 
true - which would moreover explain the bitter op- 
^position of certain "capitalist bosses". Two things 
^are possible if we do not introduce the CCP : 
either we lower the requirements and standard of the 
CAP (which is again^'c our views : we want to-obtain- 
and the trainees themselves demand it - complete 
equality between the CAP and the-^final examination 
from full-time schooling). In order to obtain this, 
« the level of the CAP must be raided ; either we 
leave 40 to 50 % of our young people v/ithout any 
vocational qualification ending In an official di- 
plorra or we introduce new stage. n 

Obviously an unqualified worker can be more 
easily exploited than a man with an official 
diploma (CCP). 

In modern vocational training there is no dis- 
tinction between the elite and the^'o'thers . There 
are only long and short periods of training. 

(II) By introducing the CCP the Ministry of Education is , 
neglecting and abandoning the cultural education of 
a large number of our young people who are equally 
entitled to general education . • 

This argument is not valid since all trainees ^ 
studying for the CCP have had at least 9 years of 
general education, 6 years of primary education 
and 5 years of complementary education in which ge- 
neral subjects occupy a large part of the timetable. 
V/e must also reject any argument insinuating that 
our primary school teachers are incapable of giving 
^ valid general education. 

Voluntary day and evening courses are available 
to all those v;ho wish to complete their general 
education. 

" "Our aim is to allow all those' with restricted - ■ - 

.powers of assimilation to obtain at least' a minimum 
vocational qualification. 

The CCP is the only way in which some children 
ol* foreign v;orkers can .obtain a vocational qualifi- 
cation because their lack of knowledge of German and 
French prevents them at f,irst from following the 
^ theoretical education. V/hen they have obtained this 
knowledge they can later take evening classes and 
obtain the CAP. 
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(lIl)The final argument and the only one which really 
worries us, concerns th^ attitude of young people 
and their tendency to choose the easier option if 
they have the choice between two methods of appren- 
ticeship. 

Some young people probably do think like this. 

It should, however, also be considered that 
the parents also have their own views and we have 
.noted that parents tend to make their children go 
in for the longest period of training leading to 
the highest qualification. There is always a gre it 
fuss if a child has to be removed from a class be 
cause hd is unable to keep up. We have therefore 
decided that trainees taking the CCP should first 
of all follow the CAP class and not be admitted to 
the CCP except if the school board so advises. 

The introduction of the CCP * thus appears a lo- 
gical conclusion for the concept applied when tech- 
nical and vocational education was reformed. It is 
purely and simply the corollary of the raising of 
'the level of the CAP, the master^s certificate and 
the -technician's certificate.* 

Other possible solutions are : 

(a) the CCP should be reserved for those who pass 
only the practical part of 'the CAP examination. 
They should however, have followed all the 
classes, including those in vocational theory, 
for three years ; 

(b ) a common basis should be introduced for all 
pupils until the end of compulsory schooling. 
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(d)GENERAL OBJECTIVES OF VOCATIONAL TRAINING FOR ADULTS 



\ ' 

The aim of thiis section is not to describe nor to 
prescribe- the general objectives of vocational training 
for any specific cdantry. It should be emphasised .that 
vocational trainingXcannot be considered in isolation. 
The general aims of Vocational training can only be for- 
mulated in the framework of the overall social and econo- 
mic goals of the country concerned. 

It follows therefore that it is not possible to 
rank or assign pr?.ority to the different aims of vocatio- 
nal training systems in the different members states. 
This ranking .can «nly be carried out once the social and 
economic goals of the country concerned are defined. 

However, it should be recognised that there are 
important economic and social similarities in our coun- 
tries and that an attempt to systematise the relationshop 
between these overall economic an^ social goals and the 
goals of vocational training may be useful. 

I shqul'd -perhaps stress that by general objectives 
I mean the ovefall.aims and objectives of the total voca- 
tional training ' system. By specific objectives on the y 
other hand/ I mean the concrete aims of a particular ^ 
training course, which are specifically concerned with 
the future employment af the trainee or with *bhe situa-- 
tion of a particular group of trainees. 

' The development of a coherent system of adult voca- 
tional training is of rather recent date. If we accept 
the need for a coherent system thi? will be . established 
-by reference to two. .factors" : 

1) the changing structure of ' the labour market ♦ 

2) technological change which on the one hand diminishes 
the number of jobs for unqualified workers and on the 
other hand, transforms the work-situation for" the 
individual worker, so that retraining becomes neces- 
sary in "order to exploit effectively new techniques, 
machines and materials. 

These factor-s which are present in all member states 
are often not only considered to be important motives 
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behind the vocational training of adults^ but indeed to 
be the onliV ones. That is, the general objectives of 
vocatl onai" training are j. 

1) to right uneraploym^nt v;hich v;ill occur if training is 
not carried out 

2) to produce a labor foi^ce which is able to carry out 
the nev; techniques called ^o^* 

With only these objectives it may be possible to 
reduce tral.ning to short practical courses with a rather 
narrow content and aiming , directly at the exist^ing em- 
ployment outlet. The fulfillment of the objectives 
mentioned, to fight unemployment and to produce a labor 
■force adapted to the new techniques is of course neces- 
sary but it may be asked whether these objectives seen 
from the view point of both the labor market and society 
are really sufficient to ensure effective vocational 
training. , 

It is generally accepted, I believe, that the chan- 
ges which are occuring ..on the labor market are not iso- 
lated phenomena but that they are, together with a numbei: 
of other phenomena, symptoms of a more general transfor- 
mation which characterises and will continue to characte- 
rise our society. For many decades v;e have experienced 
changing conditions both for the individual and^or 
society, altered economics a^nd' social rclation:»* norms 
and attitudes which seem to change over a few y-^s. 
It 'would seem neither rational nor effective to Jj^j to 
deal with the structural and technological chan^Ps which 
are taking place in the labour market without at the same 
time trying to co-ordinate these training arrangements • 
with a broader policy setling out' to give the individual-' 
as citizen and as employee the best framework possible ^ 
to enable him to live and work in a society under change. 

It is my view - which will be developed for the rest 
of my«4;^lk - that , training 'which is implemented only to 
fight unemployment and to secure the labour - force cur- 
rently^ called for, will not be able to fulfill real 
educational needs and that it will be ineffective - this 
meanf that using different but limited, resources may 
produce very much greater results. 

The real educational needs - which may not be felt 
by the trainees themselves - may be systematised under 
three headings : the vocational technical aspect : the 
moti\/ational aspect and the aspect dealing with the 
individuals relstfiQ^^^^iP to society. 
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The vocational/technical aspect is concerned with the 
fact that within a short period of time part of the labor 
force will have to learn to use new technology offered in 
a new function or in a new work situation. It is not 
possible beforehand to know what requirements the indivi- 
dual will have to fulfill and it must be accepted that we 
have only limited foresight of the technological develop- 
ments Which may occ^ur in the different sectors. 

This, of course, emphasises the need to provide 
vocational training but it also raises the major question 
of whether it might be possible to limit the demand for 
the individual to follow new courses if the training 
would impart not only the knowledge actually needed but 
also a broader knowledge of techniques and materials etc. 
within the sector in which he is or will be employed. 

Is it not reasonable to expect that such a broad 
training may make the change to another technical pro- 
cess, or to another job in the same sector more easy and 
so decrease the demand for further training ? 

Attempts should be made to build this broader charac- 
ter into vocational training programmes and I am sure 
that such programmes could be given with limited resour- 
ces if the overall vocational training system is planned 
with this aim. • ■ 

One possible solution would be that some programme - 
modules giving knowledge of the basic technology of the 
different sectors are constructed and used in combination 
with the more ■ specif ic programmes. It is important that 
these programme - modules are co-ordinated with program- 
'mes in the vocational education of the young. This may 
limit the demand for resources and in the long run create 
an effective and co-ordinated training system. 

Many research institutes and many private firms are 
working, with the motivation - problem , that is the 
problem how you can involve the workers in the life of 
the firm. We are very clever in the construction of 
effective ma c hinery, but experience has shown how 
difficult it can be to create a working situation which 
can secure our effective co-operation between workers 
and machines. ^ 

You will often trjy to solve this motivation-problem 
in two ways, either tarough job - enrichment and job- 
enlargement to create/ a concrete job -satisfaction, Of by 
various arrangements^ to give the workers influence on 
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how th^ firm is managed. 



. I have had the chance to, follow some practical ef-. 
forts of this kind in Denmark, One thing has been learnt 
from thBse efforts; that they call for much training and 
education of all the employees. You must train to impart 
the knowledge for job-rotation and for instance making 
local procedures of control. You must educate in order 

to give the necessary formation of the technique^oir* 

co-operation and for creating a positive attitude towards 
co-operation . 

If this work on the motivation problem is to develop 
rather than>>continue at the level of interesting expe- 
riments, the experiences show us that it may be a great 
help, if we, in the vocational training can train on a 
broader scale .in the technology of the sector concerned 
and if we tea^Ji subjects which .may give a background for 
a closer and more responsible co-operation in the firms. 

^^ 

As far as.-the aspect of the individual and his 
relations to society is concerned I think it is impor- 
tant"* to stres3 that the changes in societies for many 
people means a general feeling of insecurity and anxiety. 
It is well knowij^, that an anxious and nervous person very 
often will ac| (|uite irrationally and will adapt attitu- 
des which may|b^' very harmfull both to themselves and to 
others , | ^ - * 

May I point to the fact that that part of the popu- 
lation which has had the least education, very often is 
that group which has suffered bad - consequences of conti- 
nuous changes and which - naturally enough.- feels most 
insecure and anxious. 

First I should like to point to that part of the 
working force which is doing jobs requiring very few tech- 
nical qualifications: The work itself gives no possibi- 
lities for development and the lack of Vocational 
qualifications gives no possibilities for changiijg to 
another less routine job. Being employed for a long time " 
with a job of this sort will make it very difficult for 
the individual to continue to believe in his future pos- 
sibilities(;. It is for instance very often difficult to 
make these^ workers go to further training courses < 

I should liice next to point out that specific pro- 
blem which the longer and better education of the chil- 
dren to-day may raise in relation to the grown-up, 
generally 'less educated, part of the population. This 
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.problem has been called the generation gap and it may in 
long ran deepen the problems mentioned of the less 
qualified part of the working-force. The da^ may come, 
when the better educated young workers willr take the 
jc^s from the older workers - when the 'yoking generations 
wifl.1 give the. older generations a feeling, of helpless, 
inadequacy. 

The vocational training system for adults gives, 
some of the best opportunities to change this develop- 
ment, It is however, necessary, if these possibilities 
are going to be used effectively, that it is accepted 

jlthat one general objective of the vocational training 
is to try to lessen this inpact of development. The 

i task is difficult, but the aim must be to try to compen- 
sate for the lack of knowledge of their own situation 
in soc iety . 

I have mentioned some in my opinion very important 
motives for giving vocational training a broader tech-- 
nical content and at ^the same time for teaching subjects 
dealing with co-operation and with society. These 
motives must, even of the attainment of effect will no 
doubt b"^ very difficult to assess, influence the^ general 
objectives of vocational training for adults. 

Societies will Xn the future be characterised by 
continuous change on the technical/economic and social/ 
political fields ; and societies will, I believe, 
experience ever growing difficulties unless they create 
opportunities for individual personnel development - 
this being one necessary condition for the individual to 
fulfill the requirements of' changes in society. 

The pace of change is - whether we understand or 
not - a dynamic process. One condition to manage this 
process is that man gets opportunities according to his 
rich potentialities. 

-History .shows, .exampl.es. of societies which where not 

able to solve such a problem and very often the way has 
been opened for destructive forces. 

t 

Alvin Toffler quotes in his book A Future Shook an 
old worker who says "I^wish to know something about the 
future, I wish to die an educated man." 
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(e) THE MATCHING OF TRAINING WITH \ 
JOB OPPORTUNITIES -iN FRANCE^ 

Technological instraction and vocational training 
difer from academic education in that they provide 
instraction for' a specific activity. In most^ cases- this 
nude of specialization is achieved by (nore or less in- 
tense training on the job. In this ^connection, technical 
schools in France have for long set the pace. Bat 
whether training was provided in school or on the job, 
the goals of teaching programmes have so far been rela- 
tively satisfactory introduced by more or less direct 
collaboration between training schools and enterprises. 
Two recent phenomena have however called for a reapprai- 
sal of this relatively simple method of training the 
working population ^f or various occupations. 

Firstly, the . development of mass education in 
school and university in the 1960s extended the problem - 
of adapting.: training to the needs of the economy and- . " 
society. ^ 
« 

Secondly, economic growth and technical progress 
make it impossible to confine training to the reproduc- 
tion of known socio-aultural models. On "the contrary 
training must anticipate ' medium or long-term trends. 

Thus in relation to the two developing spheres, 
training and employment, the probiemof adapting training 
to employment has arisen in the Community countries, 
both as regards quantity and quality.. This problem 
which has been felt most intensely in the field of 
instruction for the professions is tending to spread 
to the entire training system for young people ^nd adults. 

This question cannot be considered without knowing- 
the objective links between training and employment, 
varied as they both are. The Centre frangais' d ^ etudes 
etde recherche s sur • les ' qualifications (CEREQ),a public 
institution with an interministerial task, responsible 
to. the Ministries of Education and Labour, has been 
working since 1971 to improve and develop a system to 
prov^ide such Information. 

Starting" with the limitations of macro-economic ana-, 
lysis in this field, an indication of the direction 
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of main research under way' to improve diagnosis could be 
useful with reference to the work of the seminar. 



I - STATISTICS AND THEIS LIMITATIONS 



The q\\&^tiV6Xi' the adaptation of training" t.o 
emplayffl^'nt was first treated from the macro - 
eQpno^4c.>angle ' -''^ . 

/'53rfi^stage is particularly well illustrated in 
. ^,fce' by the forecas'ting methods used for employment in^ 
p'reparing. the IV, V and VI Economic Equipment and D.e'v§>-^ 
lopment plans. 

Plan IV (I96O-I965) The first attempt to assess the 
' development of employment according to qualifica- 
tions. 

■t 

Tool : unofficial conversion grid using the nomenclature 
of individual activities and the six levels of qua- 
lification defined by period of studies (use of the 
work done by SVIMEZ (Association for the development 
of Northern Italy) Rome i960 ; definition of trained 
personnel requir'ed by the Italian economy ^in I975). 

Plan V (I965-I970) In I966 a statistical committee was 

set up to determine the numbers of trained perspns in 
relation to the needs of the economy. 

Tool : .a conversion grid using the nomenclature of occu- 
pations and the national vocational training-nomen- 
clature. 

Plan V'l (1970-1975) A specialised planning committee for 
vocational training was set up. 

Tool : further development of tools mention ed ' above and 

coordination with the central economic data system. 

B : The limits of this approach are ' essentially due 
to the material used and the difficulty of inter- 
preting the results which it yields. 
1. The data used on training and employment is made 
up of ' : 

a) as far as concerns employment, essentially. 




1) Population statistics which at each census give the 
' distribution of working population according to a 

nomenclature defined in 19^7, revised in 195^^ which 
uses 12000 terms for a'ctiv^cies classified under 
1000 typical occupations . 

Criticism' : The nomenclature is obsolescent and dif- / 

■ ficult to bring up to date ; 

■ individual statement's are relatively 
unreliable. ; 

the lack of any job description. 

2) The annuaT~sCnwey by the Ministry of Labour on the 

. distribution o^^employment by qualification in enter- 
prises- employ^g 10 employees or more, using a nomen- 
clature of 500 jobs ; . 

Criticism : variable field of survey ; 

i interpretation by the enterprises of a 

nomenclature based on convention. 

b) for training : 

1) Statistics on training inputs and outputs, which have 
been developed in considerable detail "over the last . 
ten years, for initial training since 19^^^^ for adult 
training since I966. 

2) Classified according to a national nomenclature dis- 
tinguishing six levels , and 47 groups of* dif f erent 
types- of training (l). 

Criti'oism : ident'if ication of inputs and outputs ac- 
cording to a formal description of trai- 
ning courses and the nomenclature used in 
•certificates 

11. A statistical comparison of trsflning resources and 
available employment only allows for a very limited 
understanding of the adequacy of training facili- 
ties (2). 



(-1) Dossier n° 8 of the Centre d*£tudes et de Recherches 
sur les Qualifications, April 197^, La Documentation 
frangai^e 

(2) Dossier n°4 of CEREQ: Les possibilites d^emploi selon 
les formations acquises dans les formations initiales^ 
June 1972, La Documentation frangaise 
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Example figures for France for 1968-I975 
(annual average) 





employment 
prospects 
(without job 
mobility^ 


.0 

qualifications 
available 
(schools and 
universities ) 




. in ^000 ' s 


% 


in'OOOO's 




Levels .1 and II 


88.5- 


14.7 


67.5 


9.9 


Levais III and IV 


158.6 


2$. 5 


" 114.4 


16.7 


Level I 


225.1 


57.5' 


505.5 


44.5 


Total 


472 . 0 




485.2 




Level VI 


127.2 


21.5 


^199.5 


29. 1 


TOTAL 


599.2 


100 


684.7 


100 



Criticism : ^ * 



(a) it is necessary to consider large aggregates, e.g. . 
the addition of resources and requirements at 
Levels IV and III for technicians and senior tech- 
nicians . ■ 

(b) it is\ impossible to take into account the phenomena 
of mobility : • 

- within the working population^, particularly from 
the point of view of job advancement and the ope- 
ration of the labour market inside enterprises ; 

- at the -time of transition from training., to employ- 
ment (utilization of acquired knowledge, level at 

• which employed, recruitment policy of firms); 

(c) it provides no information on- the relevance of the 
content of training courses to employment (knowledge 



acquired as; opposed to required). 

Thus technicians at level IV are at pi'ese 
defined with reference to the technician's dipl 
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These limitations of macro-economic analys 
are even more marked with regard to adult vocational 
training. In. this case the job opportunities depend 
on : - . 

- the decision whether to employ a young person 'just 
qualified or an-adult ; 

• - the inclusion, of a training course at an apprffc- 
^ ' pijtate stage as a real career possibility ; 

- the, at times, very delicate assessment of" the 
specialization or further training required ; 

- the ability of the adult to adapt to a^ situation 
nev; to himself and his family ; 

- the.v vulnerability of the labour market to changes 
in the economic situation. 1 

• 'But it may safely be ass^umed that diagnosis of the 
adaptation of adult vocational training programmes only 
calls for the colle'ction and processing of more complex 
data than that needed for the short guidance period of 
initial training giv n to young people. 



II - THE SEARCB-FOR NEW ANALYSIS AND 
/ ■ FORECASTING -INSTRUMENTS 

"^In France an effort has. be^n made to improve the 
links between training and employruen.t and. to dr%w con- 
clusion in the three .-foliov/ing ^way^ : improve basic 
information ; 

find methods to forecast employment trends ; 
determine teachir.g coals corresponding to occupations . 

A. improvement of basic information on actual working 
conditions and Job mobility • . ^ 

1 - Le repertoire frangais des Ernplois (l) (French 

directory of occupations). Since' I974 France has ' 



(l)^CEREQ bulletin N° l3 ' ^ A 

} • 
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been developing a national system to identify and 
describe occupations corresponding to typical jobs 
and intends to continue this in the future. Moti- 
vated by the same concern as that which produced 
the English Classification of Occupations and 
Directory of Occupational Titles, GODOT, or the 
American Dictionary of Occupational Titles, DOT, 
this effort : . ' • 

(a) involves JO. 000 surveys of businesses between 
1974 and 1978,' organized by CEREQ and carriea 
out by 9 inter-regional centres created through 
the association of the University and the 
employment offices ; 

(b) uses a system of identification common to all 
occupations, each sign making it pdssible to 
calculate the distance from one occupation to 
another ; 

(c) makes . provision for the codification and recor- 
ding by computer of the data collected ; 

(d) adds to the job description information on the 
way of access to a post^and particularly on 
training received by the holder of the post. 



- Job mobility (l ) 

(a) Up to now it has only been possible to analyze the 
phenomena of job mobility in FrarKie very roughly, 
mainly by means of a national survey called "Trai- 
ning/ qualification, advanc^ement " of a sample of 
individuals. 

(b) In oraer tc jlscover wo^-^-rint^ behaviour associated 
with vocat. 3r ;I \. -.i 1 nir... . "E, EQ : 

in the la; : : w ye3r;:> ha:^ "i:: , e a series of natio- 
nal surve:'3 c:: ^ndi^ions ^f i^cess "to employment 
after comrletior: ::f uirferent ^ypes of training 



(1) Bibliotheque . CEREQ - VL-Iuir.r n° J-* - "L' analyse 

des qualificat^ .ns et le::, cla . sif ications d^emplois- 
Septembre 1975 - ''^^ F ^^^s ^Vv^mentation frangaise - 
29,51 quai Voltair- - fo"^'^ PmS CEDEX 07. 
Bulletin n^ I6 
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relating to : skilled worket";j, technicians (l), 
senior technicians (2), students ; 

in 1975 set up -a permanent observation system of 

levels at which employment takes place and career 

paths which will gradually be extended to adult 
training courses. 

B . Review of methods of forecasting employment with 
qualifications ^ 

The aim is to situate the occupations of men and 
women in the system of constraints which goverre their 
activities. The unit observed is the socio-techni c^l 
system constituted by the enterprise which through its 
organisation determines the distribution of work and the 
tasks of the manpower employed. 

Tool : Previous models were improved and linked to the 
central economic accounts. 

B. This approach, is limited mainly by the materials used 
and the difficulty of interpreting the results pro-** 
duced. 

I. The information on training and employment consisted 
of : 

The employment structures of sub-groups and groups 
of varying size in a given economic branch (mechanical 
engineering, chemicals, construction . . . ) have been 
described. They are as far as possible associated with 
the analysis of different factors of development : tech- 
nical, social, economic. Areas of change.^are thus high- 
lighted and sufficiently defined to make possible statis- 
tical checks using surveys by sealed questionnaires sent 
to representative samples of enterprises. The occupations 
of computer and automation specialists managers ; the 
technical building trades have been^^-surveyed in, this way. 



(1 ) CEREQ dossier n° 5, "Acces* a la vie prof essionnelle- 
enseignement technologique Ibng'^ 

(2"^ CEREQ dossier n° 7, "L^Acces a la vie prof essionnelle 
a la sortie des Inst i tuts universitaires de techno- 
logie". 
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C . Aims of tralninp; in the light of the analysis of 
work situations 

It is difficult to determine the rVrationship 
between training and empl6yment , because apart from the 
difficulties of forecasting mentioned earlier, there is 
no direct of simple link between job requirements and 
training content. 

In the belief that a traihing contains just as im- 
portant an element of personal creativity and adaptation 
as does professional work, research in France has not 
adopted a determinist and inflexible approach in which 
the training programme would simply be a transfer or 
reproduction of the skills required in an occupation. 
The task which aris 3n is therefore to analyse work 
situations in such a way that the trainer may identify 
the occupational aims to which his programme and teaching 
methods should be adapted. 

In this connection the CEREQ may be considered to 
have had sufficient experience with an original method 
of observation of work to be able to undertake the next 
phase of systematic observations associated with projects 
to reform the education of young people or adults. The 
method roughly consists of the following steps : 

- in the enterprise regarded as a socio-technical system, 
to sd.ecta stage in which a project (product or service) 
is processed or prepared by an identifiable team ; 

- analyse the activities of this team ; 

- translate these activities into language ; 

' determine the knowledge brought into play in the utili- 
zation of this language. 

As shown by a study carried out jointly with the 
Association pour la Formation prof essionnelle des Adul- 
tes (AFPA), it is essential thaty the trainer, should 
collaborate in the preparation >Qnd formulation of the 
conclusions so that he can carry out his own task of defi- 
ning the educational requirements. Furthermore, it is 
clear that training programmes associated with the spe- 
cific policy of a firm must .be adapted more closely to 
that firm^s general aims. 
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However, the salient point is that this type of • ^ 
research on the skills actually used in an occupation, 
has cast direct doubt on the traditional division 
betweepf general and specialized education. Furthermore,* 
this functional, analysis of the division of tasks spe- 
cific to an organization has exploded the craft concept, 
this being all the more pronounced the higher the level 
of qualifications and- positions. 

The new instruments of knowledge being tested by 
CEREQ, go beyond a general macroeconomi c analysis and 
are intended to serve a more qualitative and selective 
training and employment policy. 

Ill - DIAGNOSIS OF IMBALANCES AND PREPARATION 
OF SELECTIVE PROGRAMMES 

Neither employment data nor economic forecasts are 
sufficient to enable the educator so to define ' qualita- 
tive and quantitative training requirements, that the 
programming and planning of training will follow more or 
lens'of their own accord. This illusion must be con- 
densed because it still continues to affect attitudes 
and even policies. 

Moreover regardless of the scope and pace of change 
in our societies not everything changes at the same t^ime . 

The primary concern of a coordinated training "a^nd 
employment policy must therefore be to prevent or correct 
the imbalances that can always arise in a free economy. 
For this reason early warning signals are important for 
those* responsible for public or private social welfare 
programmes. 

A. Definition of indicators of effectiveness of training 
on the labour market 

On the assumption that a given training is effective 
from the economical and social point of view if it gives 
normal access to ""an equivalent post, observation of the 
effects of training on the labour market takes oh impor- 
tance. Using the various surveys oh the placing of 
graduates, the CEREQ has pinpointed some significant 
indicators (l) such as : ' • , 

(1) CEREQ, Typed document, June 1974 
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- x-ate of activity at the time of the survey 

- time spent seeking first employment (immediate 
employment in less than one month, or more than six 
months ) 

- geographical mobility (percentage of activities f.ound 
outside the training region) 

- extent of correspondence between the training given 
and the post obtained • •» v 

- levels of remuneration (average by speciality, sex, 
region ; distribution by income bracket) 

- occupational classification of the first job 

- distinction between permanent employment or interim 
Jobs. 

On the basis of these preliminary analyses, a series 
indicators which are simple but significant in combina- 
tion, is being prepared, regarding the effectiveness of 
a training in relation -to actual needs. It will be 
systematically applied to successive batches of trained 
persons beginning in 1975- 



B, Preparation of selective programmes 

Whether concerned with the systematicMe'tect ion of 
imbalances between training and employment or with more 
refined studies seeking to coordinate the two trends,; 
the programmes are intended to', rein/orce^ the inst-ruments 
of economic, adjustment with^..^G)re specific ajad faf'-rea- ^ 
ching operations affecting -^the. actual structures of \ 
qual.iTicatiorjs, namely the_^^dastribution of ^kil^l^s through 
the * population . and the dis'^tfalDUtaori ,,of tasks in eriter-^ 
prises. «^,Fqur type's-^of progra'^mnias. jeap be disti^uisjied : , 

^ • -VST ■. • ■ i ' ■■ '. « , r 

1), Sector-al- programmes-" . ^ 

\ThQy ca-ll int<p, quest-ion both i nd us trials policy ^.^Ja^t • 
■is;-^tfle ais-triflbut\on o^f wo/4c associated with the ^fn- 
; ployment ^truc,tuf%^, ^^n a^ccordance with ^tj^ in^^e,rna~" 
tional divtf^^iorwof la'bour\, andj^ Strategie's'.f^jr ' - -^^ 
' , tfichn-o/l'Ogi^ai. and ^pc^ia^^-iinntPvaiJ-ions . Jhe task ^ 
vher<9^^i^V^ def/ine<| tra^fni^ -pffl^graames for available \ 
'or em^pl^jsed mdnpower^ vvhi(?fe--t*auld facilitate -transit iip,n 
- f rom ^tri^'^l^resent, sl«t^uat>ior(to\ the^ desired ^yituatipn, y 
whil^ safeguarding jobi'secutVty *and sociajr progre'^^..^ , 

, ' ' ^1' ^ / ■ '-""^ y 



."3 V -^ . ^ ^ 
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2) Regional programmes 



The difference here is that the location of enter- 
prises and the characteristics of the local popula- 
tion are an additional source of at times sporadic 
imbalances , in situations where occupational mobi li ty 
is incomplete. 

5) Programmes for particular groups 

In planning training programmes for women, young 
men and young women and physically handicapped per- 
sons, additional constraints will have to be taken 
into consideration. 

4) Programmes for enterprises 



In this case it is assumed that all the instruments 
linking training to employment are subjected to the 
company goals. 

It is clear from the foregoing ^technical analysis 
that company strategy is the key to adapting training 
to employment. 

For a long time technology and its development has 
been regarded as the prime factor in determining qualifi- 
cation requirements that are met by suitable adjustment 
in vocational training. But all the research made in 
France tends to restore technology to its true place as 
one of the factors of production. 

Certainly technological evolution has a widespread 
impact on the volume of the population employed in the 
various production sectors, by making different uses of 
qualifications depending on the machinery or processes 
utilized. But the entrepreneur is constantly having to 
consider whether if he introduces a new .'technique he will 
be able to make use of the skills of the manpower avai- 
lable to him. The widely differing conclusions reached 
in such cases under satisfactory conditions of competi- 
tion on the market show that the margin of initiative 
is unexpectedly wide. This aspect of the problem deser- 
ves attention at a time when both the qualifications of 
the population have evolved considerably as a result of 
changes in the educational system and may be still 
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further changed through lifelong education, and when 
the constraints on our production system are increasing 
the pressure to develop technical and organizational 
solutions to labour problems which will make the best 
of the skills and abilities that a population or nation 
*fnay have or may acquire. 
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(r) TRAINING FOR VERSATILITY 



' - , The project concerns the training of young sewing 
machinists in the clothing industry. 

Due to changes in market conditions, the lengths 
of production runs is decreasing and the variety of 
styles is inci'easing. 

This 'creates needs for different kinds of skills 
to those developed by conventional, training systems. 

These demands are imposing strains on organisations 
and are believed to be largely responsible for highe^ 
labour turnover, recruitment dif f icultieSj^<- increasinW^>o^ 
labour costs - and problems of work organisation. 
also implies that conventional training has lost \ 
validity. 

These problems highlight the need for re-defined 
training objectives and methods and, above all, for a 
soundly based internal evaluation of training. 

Part of .-^he background to this project is more 
theoretical, iBut is vital to the conception of the pro-^ 
blem. Our- experiences have led us to assume that certain 
aspects of the organisational climate are likely to 
affect the development and utilisation of versatility 
in the labour force. Furthermore, it appears that these 
climatic factors may be more potent than either the sim- 
ple effects of a training scheme or even the particular 
form of production organisation. The evaluation of 
training therefore cannot be designed without these 
factors being taken into account. 

The Project ^ ' 

After some preliminary development of training 
■ideas, training schemes of comparable designs were im- 
plemented in two different clothing firms. The broad ^ 
objectives were to develop versatility and self-organi- 
sation of work groups. 

The most important differences between the firms 
were : 
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Firm A 



Firm B 



Location 
Size 
Variety 
Product 

Management style 

General ability 
of recruits 



Rural 
Small 
High 

Ladies fashion 
garments 

Adaptive 

High 



Urban 
Medium 
Medium - 

Chil irens outer 
garments 

Inflexible 

Average 



In addition to developing the operator training 
scheme, the project also had the following purposes : 

(1) To validate predictors of versatility and group- 
worlcing potential for the purposes of selection. 

(2) To re-define the. role of supervision so as to 
facilitate maximum development of group autonomy. 

(5) To develop courses for instructresses and super- 
visors to prepare them for their re-defined roles, 

(4) To develap the broades^ methods of assessment of 
production performance . ' * 



Operator Training 



The learning objectives of the operator training 
scheme were : ^ 

At the end^-of the course, trainees should have the 
willingness and ability to - ; 

. cope with a variety of unfamiliar operat-ion^. , 

. require decreasing- periods of re-training 

. minimise production losses due to change over to 
new Operations 



work within a self -organising production group 
of machinists. 
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The assumptions underlying the training design were : 

. the basic skill of versatility concerns the 
development of favourable attitudes to work, 
group-working and self -organised learning 

. the learning enviro^nment must facilitate and 
support trainees in the development of thise 
attitudes ♦ 

« learning obj ecti ves must be clearly stated for 
each operation to be mastered 

. the style of instruction must be Learner orien- 
tated and have "counselling" characteristics. 

Instructress Training 

•The objectives of the instructress/supervisor trai- 
ning dourse were : 

(1) -To promote the establishment of "counselling relation- 

ships" with trainees. 

(2) To promote the introduction of opportunities for 
"self -organisation" , 

The training method is based on the systematic 
examination of "Training Events" by instructresses and 
supervisors, the hypothesis being that, by examining 
these products of their own attitudes, they would be 
able to see the possibility of alternative attitudes. 
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CHAPTER 3 



EVALUATION WITHIN THE TRAINING 
PROGRAMME 



Introd 



(a) An Overview 



In the previous chapter the various factors - social 
and economic policy, manpower objectives - which result 
in the decision to implement a training programme were 
reviewed* 



In this chapter we shall deal with the various ele- 
ments which should t!e taken into account in designing 
and evaluating the programme itself ; there exists a 
comprehensive literature on teaching theory from the 
field of education, on task analysis from the field or 
technology and on testing from the fields of psychology 
and statistics. The problem- a recurrent one in the eva- 
luation of training^is to relate the insights gained 
from other disciplines to the specific situation of the 
training programme and to li^k up these different ele- 
ments in a systematic. and manageable^way . ■ 

Once again we introduce the material by means of a 
diagram (2). This is an expansion of Ifhe innermost 
elements of diagram 1 (p.l6). Ifi this section the evalua- 
ti'on of the more important elements will be dealt with 
in turn : - 

- Task analysis 

- The setting of learning, objectives % ■ 

- The assessment of trainees. 

■ - The measurement of what has been learnt 

- The application of teaching theory. 

Diagram 2 is of course only one possible cc^'^ ptual 
framework within which the different elements can "oe 
loc;ated. During the seminar held in Mancr.^ster or.e of 
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Diagram 2 



' Outline of the elements involved if) the design and evaluation of a training progrannme. 



Task Analysis 



Learning objectives 



Irriplement Teaching 



Test Learning 



Test Task 
Performance 



Assessment of Trainees 



Learning/Teaching 
Theory 
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the working gr-oup:.; adopted its own framework which is 
of considerable interer.t and is annexed on p. 68. 

The material in this section will be found to be 
fairly theoretical and not easily digestible at a first 
reading. The annexed papers howevei' const itutt^ clear 
practical' examples of some of the issues dealt with in 
the text. In part icular , sect ion (c) may serve as a 
useful simplified introduction to this chapter. This 
paper gives practical advice on how to evaluate a tra-p.- 
ning programme and examples of questionnaires used by 
a group of Dutch industi'ialists to evaluate their inter- 
nal training programmes. ^ 

* S 

Task Analysis ; Problems and methods 

Due to the degree of skille;i<^ef f ort required for 
adequate task analysis, it is Tiot uncommon to find a' 
training programme based on an analysis which was car-; 
ried out in the past but has been superseded by event? . 
This tendency is frequently re-inf ojr^ced by a desire to 
economise on training plant, thus leaving traijiees to 
master skills appropriate to machinery no longer in 
productive use'. 

It often occurs that the task to be analysed is not 
just that of one type of operative working with one type 
of machine but rather the task of a group or category of 
workers. The paper by Dr. Kemp shows the complexity 
involved in carrying out a task analysis on "industrial 
master-craftsmen". The task analysis is clearly shown 
to be the prerequisite of a system of examination and of 
certification and hence of access to Jobs. What is at 
stake, therefore, ^as the paper points out is not just 
the technical problems caused by task analysis but also 
the problem of reconciling strong professional and eco- 
nomic interests. 

There is also ar^ increased interest in , adaptability 
or versatility as an objective of^ training. These compe 
tencies have r-eceived formal recognition but lit^tle 
practical attention either in training or in the educa- 
tional world.- Consequently there is little in the way 
of formal , evaluation technique with respect to such fac- 
tors. Additionally, the organisational milieu in which 
.they are developed is of crucial importance and this 
wider perspective must therefore be included in'the, 
evaluation gf training programmes. 
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J Analysing a task can be either a selective or a 
comprehensive pre 'j'iure. F^or manual tasks an exhaustive 
description known aS "Skills Analysis" (1) is frequently 
used. This entails a detailed charting of each hand 
or. foot movement and the sense modalities by which they 
are controlled . Thi s type of exhaustive analys is . i s 
now frequently replaced by a "critical incident" approach 
which is designed to select only those factors which ma- 
ke the difference between a good and. a bad performance. 
These are revealed by comparative observation of^ good 
and bad performers - particularly learners. Apart from 
being more economical than exhaustive description, it 
is more wi-dely , applicable,- being suitable for. managerial 
and manual skills alike. 

Setting learning objectives 

A statement of learning objectives is useful inso- 
far as it provides a clear criterion against which to 
assess learners or provides a goal for learners and 
trainers to aim at. The modern tendency is to write * 
"behavioural" objectives. To qualify as "behavioural" 
an objective should state 

(i) What the ^eaVnen has to do. 

(ii) Under what conditions. 

(iii) To v;hat standards. - v'. 

(c.f . Mager(2) . 

This approach derives mainly from training in tech- 
nical skills an* is mast suited to that form. Thq gain 
in' precision between the following two exampl^es is 
obvious : . " 

(a) Nan-behavioural - "Learn to do simple millinig 

operations" 

(b) Behavioural - Learn to : 



(1) Seymour V/.D. "Training for Manual Operations" Pitman. 

(2) Mager "Setting Education Objectives". 
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(i) Perform operationsx, y, z. 

(ii) Using metals q, r, s. 

(iii) To a specified dimensional tolerance and 
a specified quality of finish. 

Unfortunately, trie validity of this type of objec- 
tive has been overgeneralised . It is not always 
realistic to try to be so precise about e.g. supervi- 
sory training.^ In this case objectives need to be 
written so as to focus attention on the topics to be 
engaged, rather than as^ hard criteria. 

The derivation of learning objectives from a task 
analysis is via the assessment of the learner's initial 
competence. This step is frequently omitted, with the 
consequence that the 'training programme is burdened with 
unnecessary material in which the "learner^ is .Iready 
competent. To an extent, this is inevitable' when groups 
of learners have to be taken as they come. But where 
possible the programme should cater for significant 
differences between learners by classified entry and/or 
by adapti-ve programming which permits ^aried starting 
points (and varied F^a-ths ) . A fixed training period- may 
be a reflection dhly of fixed ideas and an evaluator 
should watch careful).y for the uncritical implementation 
of a task analysis "as a set of -learning objectives. 

At this stage it has alscTto be . .embered that the 
analysis of a task may ttjell yield crucial elements which 
require action other than training. T.F. Gilbert has 
coined the maxim "if he could do it to save his life, \ 
it is a performance problem rather than a training pro- 
blem". The distinction here between Inclination and 
ability. There is certainly a valid point in that 
certain aspects of performance may best be facilitated 
by incentives, job re-structuring, equipment- re-design 
etc. Reference has" already been made to the importance 
of the organisational milieu both in defining the perfor- 
mance required and in sustaining it. However the surface 
distinction between training/non-training is misleading 
and implies a narrow view or training itself. Inclina- 
tions are also matters of learning, but noX necessarily 
of pure technical instruction. It is. necessary therefore 
,to make distinctions between objectives best served by 
technical instruction, those best ^«rved by other kinds 
'of learning and those for which alteration of work JTon- 
ditions is most suitable. Using the terminology of 
J. Raven's paper (Ch.2, section b) we could express the 
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distinctions as being between : 

technical competencies 
personal competencies/attitudes 
working conditions . 

A common feature of training programmes is that 
they cater for the first, demand the second and ignore 
the third. 

The paper by M. Pearn "Wider Opportunities for 
Unemoloyed Addlts" (section e) describes a training 
courjfee whose • objective is not to give skills training 
as such^ but is rathe^?- aimed at ensuring that when the 
trainee leaves he should "^V,^ ' 

(a) be motivated to seeK employment 

(b) be familiar with the work outlets in his area and 
be realistic in his choice of jobs 

(c) know how to seek, apply for and hold a job 

3 learn to use his initiative to assess the needs of 
a job and lear'n to perform it successfully. 
ASSESSMEOT OF TRAINEES 
es of Test 

Selection tests are frequently used and sometimes 
used well. Faulty technique rji this area can reasona- 
bly be held responsible fjpr a^large amount of waste - 
both of training effort a\d of ability. 

Standard tests fall into two main groups : tests of 
ability and aptitude ; and testa of values such as 
attitudes cr interests. (The term "test" for this latter 
type of measure is particularly misleading since it 
suggests "passing" and "failing", which is inappropriate" 
in terms of attitudes). Ability tests themselves 

sub-divide into attainment tests, which measure a 
person\s developed competence in a particular area (e.g. 
trade t^st in car mechanics) and aptitude tests wh_ich 
■are designed to estirrvate a person^s potential to become 
competent (e.g. tests of mechanical aptitude). Intelli- 
ge:.je tests attempt to estimate a general aptitude over 
a wide rang^^ of cognitive tasks. 
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Fig. 3 : Types of Tests 

^^^^^ Tests 
Ability ' Value 



Attainment - Aptitude Personality Interest 

profiles if^r^r^r'^^ 
Attitude 

Inventories 



Interest inventories are generally used in gui- 
dance, though some tentative use has been made of them 
in the field of selection. 

Use of Tests 

Because of the direct relation between training 
content and test content, the attainment test is the 
simplest to use. Judgements based on aptitude tests 
areJiriore prone to error since correlations between trai- 
ning performancB and test score are often surprising 
and unpredictable. The complexity of the relation 
between test ft :tors and training performance is even 
more pronounced in the case of ' personality profiles or 
attitude inventories. 

Whichever kind of test is used, certain libasic 
procedures are necessary : ' ' y 

1 - Before any testing is done,. a rel iable Xnd valid 

criterion of task performance m .ist be Established. 
A valid test is one which is appropriate-one which 
in fact measures what it is supposed'^to measure. 
Without this, testing is no more than a supersti- 
tious ritual. ^ 

2 - Before tests are used to select trainees, trainees 

must be used to" select the tests. For an initial 
period, all those tested must be accepted, so as to 
provide the necessary criterion data by which jbo 
validate the tests. The establishment of test- 
criterion relationships is rarely straightforward 
and usually requires expert assistance. 
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3 - Any change in trainee intake, or in the training^ 
programme is liable to upset the operation of 

selection tests . Periodic checks are advisable . 

/ 

Without the kind^^of data referred to here the ef- 
fect of testing on a training programme is as likely to 
be zero or negative as to be positive. 

s 

The underlying rationale of testing is found in the 
experimental method used in the physical sciences. A 
short note on the assumptions underlying such research 
and their application to the evaluation of training is 
on p, 11^. 

The interview compomenC of a selection programme 
can be validated in the same manner as formal tests. 

Tests of Learning 

Follov;ing general practice we can divide learning 
into three general aspects : 

Cognitive (intellectual, knowing about things or 
systems). 

Psychomotor (muscle skills). 

Affective (attitudes, values). 

Whichever type of learning is involved, the testing 
of v;hat has^ been learnt is a crucial element in the . 
training process. It is determined by the initial task 
analysis and selection of training objectives. If for 
any reason the testing of what has been learnt is invalid 
(inappropriate) or unreliable (subject to irrelevant 
variation), the operational efi'^ectiveness of the training 
will be severely impaired. The teaching process will be 
deprived of some important feedback, quite apart from the 
effect on decisions about trainees and their subsequent 
Jo:)S. 

C o gi:i i t i v^e skills 

For cognitive tasks, the translation of a task ana- 
lysis into learning objectives can prove problematic . An 
analysis shows only what is done in a given range of 
situations. That is, it is concerned primarily with the 
surface structure of behaviour rather than with the deep 
structure (i.e. understanding) .' 
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xlfvall ^h^t is /leeded is the .reGall of, say, tech- 
niclft- i^f Pirmationv^ ^Jme* s^i will be a su-f,^ , 

.ir|cie'r>t/.sy^-de. It/l|j^wHen a 'more generalised^- abilit/^' v. 
is requi^d^thali the ' -{^uvf ace structure becom^^ ifiadequ'^*^ 
bf Itself . - • '/ ' ^ ■ •. 

\ ^.''•Two common prtebl'emB )ih this^ares^ arfe 

- tests whicH' are oJ/erbuTdened with iti^Jgs: aemandinif .thq. 
recall »of - formula- eta ■ *^ ' ••C'5^^'^-''"-i-' ' * -j-'^ 

- theory .tests at whichjrtrost tr>aine'e|^;^u, bacTlV'. ■ 
.connection with this .fatter cese, vj^t; *ie, wellV-noted /^i^ 
that there' is 'a tend^enSby 'fc^r ^^ejhplc'ai ins.t-Kactprs ^^o 
recapitula^te -the^theopyv'gi'ii^e^' ij^-^t^eil- ^Qj^^ical. 
training whether oV'^not this is>-appropriate|^^:» the ^ 
training\ task' rrt . hand . ' - ■ '^^ 



Tests emphasising recall, of in^oi'mation are npt ' 
necessarily inappropriate - .the ; o.b^ective may simpl^'^" 
require th^t . But a ' severe- Ibad on 'memorising is fr;^.*^>;l 
quently indicative of sotnethlng. wrong in the task analy-^ 
sis or the setting of learning. obJecLives . This is 
particularly important since, not only does a mevmorising 
test fail to assess the deep structure (understanding), 
but there are theoretical grounds for supposing that it 
may well be negatively related. Thus a trainee scoring 
highly on such a test may in some cases be ' demonstrating 
that he has assimilated little. 



Affective Learning 



Testing for learning -in this field is notoriously 
the most difficult and elusive *of' all kinds of psycholo- 
gical assessment. Any instrument such as an attitude 
inventory on personality profile *can be faked to somq 
degree, and even if it is ans\nered sincerely is likely 
to be unreliable. Partly this is due to short-term 
fluctuations in the way a person feels, and partly to 
ambiguity in the questions he is asked. However care- 
fully" constructed a question is, there always remains 
the possibility that the question the person is answering 
is different from the question ;the test-writer thinks he 
is asking. Added to this is the fact that people are not 
always av;are of their own implicit attitudes or values. 
Thus the information which may be obtained is limited to 
what the respondent' is willing and able to give, and even 
*then it may change over a short period oT time. Finally, 
the answers as given have no absolute significaMe, 
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being suocoptibie of , differing interpretations . There 
are available GtanLiarciisod questionnaires which overcome 
some of these difficulties through extehsive validation 
studies. But these i^till require careful interpretation 
by a person v;ith adequate experience. 

Because of thi^sc ,-ii f f iculties ^ observational data 
is frequently usc.i. Here^ attitudes etc. are defined - 
in behtivioural tei'rns ^nd the person is observed (by 
himself or others) v%!iile actually at work. This yields 
a behavioural check on underlying attitudes or values ^ 
but "behavioLU^al" should v.ot be confused with "objective" 
(in the sense' of "non-subjective"). The assignment of 
meaning to behaviour is still.a tentative^ interpretative 
process. The question of aff'Ictive learning tests is a 
matter largely of what is not knov;n and an expert is 
expert only insofar as he entertains adequate doubts. 

At the practical levels v;hat can be emphasised is 
that any procedure which is in use as a test of "'at ti'tude 
development etc. should itself be subject to well desi- 
gned validation. Thera is a tendency for some trainers.. ' 
to use such tests with enthusiasm rather than discrimi-' 
nation. Since the ethical questions about such tests 
are sensitive ones\, it is not only a matter of operatio- 
nal efficiency v;hich is at^ stake. 



Selection of teaching methods 

Despite the controversial state of learning theory^ 
there are available a number of practical guidelines to 
the reiationshJ. p between methods of teaching and various 
kinds of learning objectives. However teaching methodo- 
logy is not an exact science - if it is even a science 
at all. The guidelines discussed are offered as a basis 
for thinking about method^ not' as ready-made criteria 
for' instant application. 

Teaching for psychorr^otor skills 

These skills are characterised by their "private- 
ness". The way a craftmen's muscles respond to cues of 
different kinds is not easy to communicate verbally.- 
The words do not exists probably for the very reason 
that a skilled person cannot bring to consciousness how 
he achieves. his results. But^ surrounding a craft-skill 
of this kind there is dlv/ays a matrix of knowledge which 
certainly can be communicated by^ symbolic means. The 

^# 
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eff ectivenesc of training i'S'- severely impaired if the 
distinction between these tv/C- aspects of craft perfor- 
mance is lost. 

On one side, it is futile to expend time in tel- 
lingj explaining and showing a trainee how to do 
something which he can only learn by doing it himself. 
The emphasis must be on practical exercises, and these 
should generally be graded (a matter for careful expe- 
rimentation) . The type of mistake referred to here 
frequently shows up as irritation or loss of temper on 
the part of the trainer. Frustrated by his impossible 
task he is led to blame the trainee for "not paying 
attention", "not concentrating", "not listening" etc. 

The corresponding error - of treating job-knowledge 
as a psychomotor skill leads to equivalent waste. Using 
pr^actical exercises to transmit what can be told or 
demonstrated is an uneconomical use of time. For ins- 
tance, in teaching the use of machine tools, lengthy 
exercises involvi ng. the cutting of metal are often 
used to transmit ideas about setting up work which can 
well be embodied in exercises with diagrams c5r models. 

Cognitive Skills 

The crucial distinction here is, between rote lear- 
ning and meaningful learning.. ^ - 

- Rote learning is, 'like psychomotor learning, largely 
a matter of practice. The important ■ variable affecting 
the efficiency of learningJis the provision of immediate 
feedback. This has the effect of preventing the consoli- 
dation of errors which are frequently difficult to -un- 
learn once they are established. Lecturing is thus 
generally a, poor method of teaching for this type of • 
material. Written instruction is better in that i\ per- 
mits selfpaced work, which is of considerable importance 
for adult learners. Programmed instruction is particu- 
larly valuable since it has also built-in testing and 
feedback . 

More important is meaningful learning, i.e. learning 
to understand theories, principles, systems or processes. 

Errors are commonly the pivot of learning about 
ideas. The concentration of "educational 'technology" 
on error elimination by the use of carefully programmed 
small steps has benefitted that kind of training where 
the task is mainly one of assimilation of new data. 
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Hov/ever, where lear-ners need to accomodate to new con- ^ 
cepts, the imkirig of errors provides focus and stimulus 
to the learRng process. The treatment of error-making 
i^ in fact a key characteristic of training method. To 
be effective, errors should constitute a genu4ne problem 
of the learner's own making. This implies the use of 
problems and exercises which connect, with the learner's 
existing knowledge suf I'icient ly for him to feel, able to 
produce a reasonable answer, though not necessarily the 
correct one. This approach tends to minimise lengthy 
verbal explanation which is in any case unsuitable for 
many adults. 

By avoiding these difficulties, the method may in 
fact yield a direct saving in time. The process of 
devising appropriate problem sequences is also a strin- 
gent discipline on the training designer and encourages 
the elimination of superfluous material. Against^his 
it is true that learning by problem solving may tal^©«^ore 
. time than it takes to cover the same materiel by direct 
tellin^:. 

The quality of learning that is required will deter- 
mine the choice. * 

The assessment of cognitive teaching method can be 
•represented as a checklist : 

• 1 - Do the learning objectives imply meaningful or 
rote learning ? 

2 - If meaningful, is the learning mainly a matter 

of assimilation of data within an existing 
structure of ideas or accomodation of ideas to 
a significant new form ? 

3 - Are errors minimised or utilised ? 

4 - Ar^e errors produced by genuine attempts at 

problem-soving by the trainee ? 

5 - What is the ' previous competence of the learners 

in the subject ? 

6 - How old are the learners ? 

7 - What is the level of memory-lead • imposed by the 

training methods ? 



An archetypal "bad'^ training method would yield the 
following results : ^ , " 

."Learners in middle life, approaching "a new subject 
with difficult ideas are given ' lectures .or tech- 
nical notes designed to enable them to produce 
correct answers straight away". 

Affective. learning 

The approach to attitude development is very va- 
riable, each trainer using his own personal criteria of 
success or failure. 

As a general coijiment it can be said that significant 
learning in the affective sphere is a^lways of an accom- 
modative rather than an assimilative nature. Thus it 
implies personal discovery or insight rather than 
"being told". Apparent changes can be brought about by 
direct persuasion, but they tend to be superficial and to 
require a supporting enyironment to sustain them. 

The analogy with the problem solving method in 
cognitive learning is close. The essential process is 
the generation of problems by the-..learner himself as the 
focus and pivot of learning. The process is likely to 
be even more difficult to manage than its cognitive 
counterpart since the problems' -are less clearly related 
to "public" phenome^^, being more unique to the indivi- 
dual. The very uniqueness of what is being learned 
makes it impossible to specify in any detail how such 
learning should generally proceed. However, it is per- 
haps possible to* pick out a few' external characteristics 
as being typical of successful learning/teaching 
situations. 

The important dimensions of such learning situa- 
tions are : • 

- the self -generated nature of the learning material, 
-the degree of control of the process by the learner. 

- the teacher's, role as animator and facilitator of^ the 
learning process . 

- the balance^ gf challenge and support. 

In relation to affective learning therefore, objec- 
tives can only be indicative, not definitive. The" • 
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learning objectives serve to indicate the area in which 
the trainer may choose to challenge the class - and in 
wnich they may challenge him 7 but they cannot define 
hov-; Che challenge will be- taken up or with^what final 
result. . ^ 

As a natural consequence of these conditions, the 
evaluatbr's attention is inevitably shifted away from 
the specific content towards the processes in training. 
This "is appropriate not only to train ag courses where 
the objectives are predominantly and explicitly affec- 
tive in nature, but also to training courses generally. 
Behing many apparently specific, technical skills are 
personal abilities such as autonomy, judgement etc. 
which are the products more of certain kinds of training 
than of particular training content. The recognition 
of this is an important change in .the modern conception 
of learning and training. 



(b) RKPORT OP A SEMINAR WORKING GROUP (B?) 



le groups adopted a model proposed by one of their 
members, Mr. P.J. Edney.. This model was seen as an ac- 
ceptable gener^al spec^Tfi cat i-on which, while capable of 
bein^ highly specific in situations where adequate 
bas^ine information was available, also had the advari- 
ta/ge of handling, though in cruder fashion, training 
schemes where the necessary data base was less quanti- 
fiable. 



The following is an outline of the model. 



Original 
operational 
System 1 . 



Training 
Criticality . 
Analysis 5 
.-1^ 



Interface 
_> Analysis 4 



Modifier <■ 



Env-iroment 
^ 

Simulation 4 — Learner 



C ID 
\/ CO 

Informal'^ g Formal 
Eval 6 ^ Eval 6A 

Eh 



Sub-Evaluation ^ 
System 7 <:. 



Changed Operational 
System 2 



The basic assumptions are that : 

(a) Evaluation needs to have a base line from which ■ 
measurements or judgements can be made. 

(ij) Training is a change agent taut£^^ there are many 

• others - e.g. Organisational Development, incent^'ve 
schemes. 

(c) The setting of the base line is an integral part of 
evaluation and is a continuous process throughout 
training. . 

\ 

■ 00 ^ 
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certain essential functions emerge in an evaluation 
system. These are described below and- refer to the num- 
bered components in the diagram above. 



It is necessary to describe in as precise terms as 
possible the components and behaviour of the system 
before training is carried out. This system may be a 
group of students prior to training, a factory, or a 
school system possessing many interlocking occupational 
groups, seen as a totality. 

( 2 ) ^Changed Operational System 



This represents ttie system of behaviour that is 
needed after training. It will be described in exactly 
the same terms as the original system and provides the 
"target" of training. 

(3 ) Training Criticality Analysis 

The 'function of this component is to identify the 
degreSaPf behavioural change, and the nature of the 
chan^^^equired between System 1 & 2. 

c 

In its refined state, it < 

(a) Quantifies the contribution that each element 

of change (through training) will make^ to Sys-tem 2 

(b) identifies those elements of .change which will 
not produce s ignif icant contributions to opera- 
tional effectiveness 

(c) enables the effects of each change to. be* seen -in 
production or f inane ial^t<erms if required 

identifies those changes that are more effectively 
produced by other change agents. 

Some parameters used in 'the computation of this 
analysis would include : • ■ 

(a) The size of change, required in each element., 

'(b) The frequency v. oh which any change will be used 
by the new system. , 

(c) Weighted date in cas^ of special requirements - 
•i.e. safety, error risks, etc.. 



( 1 ) Original Operational System 



/ 



/ 
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(d) Weighted data .showing elements which, although 
srtall, p^"e cr-itical to the operation of many 
oth;ers 

(4 ) Interface Analysis 

The function of this component i/S to translate into, 
ideally, behavioural training objectives those elements 
of change which have shown themselve^^ ,to be highly 
critical to the operational ef f ect i^^^i^ess of System 2, 
The analysis also considers the int'^r^ction of the ele- 
ments in practice, a&d the interaction of various 
occupational groups -for example, the definition of 
training objectives for management would be considered 
in the light of the training obje.. "^ives for "hose that , 
are managed,^ and vice versa. 

(5) Simulation • • ' 

This defines the most effective specification ''b.f 

training needed to achieve the objectives, both in 

learning a:.i cost terms. It uses three major compo- ./ 
nents . ^ 

(a) The enviromental factors under which the training 
will take place 

(b) The characteristics of the cpntent of learning 
. (c) The characteristics of the learners. v-^ 

Sophisticates models can provide a very accurate 
def initi-on of the learning process h^(\ include time - to 
learn, both group and individual opiT^inurf. t.i.me-tabling. , 
effective use of facilities, etc. nd aefiniLion of 
longitudinal or modular courses, :;eaching -nethods etc. 

.( 6 ) Inf.ormal Evalua^tion ^ 

<« ^ 

Enables the quality of each element of learning to 
be evaluated as it is designed . 

(6a ) Formal Evaluation 

Evaluates the' synthesis of the individual elements 
of learning into, cohesive groups - or ^courses. 
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(7) Sub-Evaluation System 

This provider the essential sensitive lir^between 
the training System and the new operational s^^em. It 
does this by 

(a) Receiving continuous feedback from the' tj^aining^ 
course in order to assess whether it is continuing 
to Achieve the objectives. 

■ ' ' • \ \- - • 

(b) - Receiving continuous feedback from the-^new opera- ' 

'tional system (System 2) in' order to evaluate the 
'validity of the changes 'produced by the 'training 
^ystem, and to assess whether •''the criticaiity 
analysis (Component 3) has been, and continues 
to be effe.ttive. From its finding* it uses the ^' 
mocj>fier feo change the working of . either Compo- 
nent 3. 4. 5. or 6. ■ 
• Therefore the -evaluation system employs 5 compo- 
nents =■ Nos. 3^ 5, 6, 6a and 7, . ^--^ 



Problemj ^ ' . , 

, 1) The -model we have 'considered was devised in order 
to deal ef fectively -With' the training of large 
occupational gr^Qups.and also with- total systems ' ^ 
such as factor$;PS'". whe^e a- wide variety of occupa- 
tional Groups v/as.' recognised, as a |ingle system. ^ 

Though we believe the^Jiiod'el can'b^^^sed to deal' with 
p'roblems relating to, for ;example - 

training for public services 

training oi* the unemployed ' 

training of redundant workers in nei^ skills 
■ v '^^training" Tor upward' mobility, . - .. , 

it obviously needs to be studied' so. t^hat appropriate 
techniques for d^ealing with such problems can be 
devised, within its conceptual framework.^ 

2) Repeated reference was made to 'the lack of^^adequate 
measuring instruments for ■ determining the 'hun>an fac- 
tor" elements such as ' attitudes , Motivation, ability 
to co-operate, and so on. 'It is recommended that:/ 
a'agi'opriate research and pilot studies for the dever 
^ "^^^fhent of such instruments be undertaken. ^ 

■ ■ ( ' ■ . 



. (o) DOCUMENTATJON Atg) EVALUATION, JDF COURSES 



EVALU ATI (jpSif .^|E1nIERAL 



IMTRODUCTIOM 



P RODU CT*;- :A Np BJ^CESS 
-SJ — ~ 



••I- 

LUATIOg 



J 



"Ev',^luation''- when applied, to education means : 
appreciating tra'Ining proglrammes with a /view to making 
well f ounde'dMecisions possible. 1^6r example : such a 
decision' could be to make i^^ip^ovements in a course. 

The raining process "may be regarded as a production 
process,^ the product t),eing. ,^:he ef Tec ts of learning. The 
measurement, of thet^e. iearnirig effects is sometimes callef 



PRODUCT EVALUATION. 




instruments , such 
'^e measures of the 



Jr , For a long tirn^'*- evaluation in graining remained 
riniitred to this kina;^of' evaluation.. ;. Great efforts were 
put; into Vie velopin^ %G.G.urat^ . 
as tests which v;ere to. ■provide 
effects of- learning. 

Gradually ■ t]rj,.e cony^ctibn Uv^s'^^^kn that simply 
checking^ the results pj^^/learning v;as not enough. If one 
wants better tra,i-.hing ' programiries ^ it is also-necessary 
to include the'' JeiirinirigW^^ process into the pic- 

tteep For it IS - inj^ tha-t . very-', process that the causes of 
wl^ learning eff ^cts_ rti^st^ be/' sought . In PROCESS 
EVALUATION, ..th'en^'^ a II''^. 'aisjiip"^ the learning- teaching 

process ar*e .ey.klviii\^p.lhy^<.^'.^-^ . ■ ' - ' 

What; P'l'?^^*"'^---^" product evaluation 

w.itltin .the traini^'^'- this consider the 
following sirap:le:^bde;I-0-;; V- ■. ' 



knoi^ ledge 
'Objectives 



r^y^ufeh' 

d'e?ryt^gn ■ 



7pr 



drawing up 

the . 
pf'Ogramme 





4 ■ 




5 




giving 




evalua- 




■ instruc- 


7 


tion 




tions 







Boxes 1,. 2., ana''3 together form the stage of curri- 
culjam develooment. ^ Box 4 is the actual execution of the 
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"training activity. The instruction is followed by an ,^ 
inquiry into the learning effects, e.g. an achievement" . 
test, which belongs to box 5« ' '-X' u 



However, box 5 -implies more than this pr^uct 
evaluation. For.ifcthe results are not satisfactory* 
one will ha^^e to look for «^ explanation in' the way 
instruction was given;. This again may bring to ligh;t ^ . 
what went wrong, in box 5. If great shortcomings are 
found, one may have to revise the main outlines f rom ■ .. 
box 2. Then again insuf f^icient learning- effects may 
traced back to a wi;ong estimate of the students' entrjr.' ^ 
knowledge (box l).- /''^ 

Finally, one may also have gained insight into the 
feasibility of the objectives, their motivating power 
for' the students, their rel^evance foi* -real life, vet. c-^. 
It is very well possible', even probable that the t^fc^Jec- 
tives will have to be revised. ■ 
' ' \'- 

EVALUATION ON A SMALL SCALE ' ' ', i 

In the/iptroduc tion we touched upon trie' .-evaluation'^ 
.of the entFr^ ,^earninarteaching process, th^t^ is, eva- 
luation on -a' large scale. 

^; t 

We would like to point out that on a sm^ll scale, ^ 
too, evaluation must continually take place.. Here the 
activities repeatedly run through the same cycle. , 



determine* 
s t-irtihg point 

and * 
f i n Lshing point 



^ execute process 



evaluate 



'Every process is prr*eceded by a reflection on where 
to start and what one wants to aj;?hieve, ,, In'^the same way . 
every process must' be closed of f. with 'a consideration of 
how things went, whether the initial situation was cOr- 
rect-Iy estimated and ^^ther«-;the/'ob jectives were reached. 
* / 



The design of a le 



on what the students al 
to be brought acr'oss to 




is preceded by a- reflection 
know and on what is supposed 
effl. Once the design is 
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finished, one must go over the' result to see if the 
presumed entry knowledge will be sufficient to learn the 
subject matter ; or to see 1-^ not too much subject mat- 
ter is packed into this one lesson ; etc. This 
immediate check is not only important for improving the 
lesson, but also useful for future lessons to be desi- 
gned. • 

The actual giving of instruction should be pf'eceded 
by a preparation. The teacher ^akes up his mind about 
how to check .up on the required entry knowledge and how 
to make sure ,that the objectives specified can be attai- 
ned in the^vtime available. 

After the lesson he -should evaluate whether the 
required entry knowledge was actually present, whether 
all subjects were treated in order, whether 'his way of 
treating them was acceptable, etc. This^ immediate check 
is not only important if the lesson is to be given a 
second time. .It is also relevant for later lessons when 
subjects that were omitted in this lesson will have to 
be treated in full. 

In a similar way the cycle applies to : 

designing a series of lessons ^ 

giving a series of lessons 

- drawing up the main outline of the way a subject 
will be treated , 

*^ 

teaching a subject in the course of a scholastic 
year 

etc. etc. 

WHY IS EVALUATION NECESSARY ? 

One may wonder : why evaluate ? Is it not a 
luxury ? Or in some way superfluous ? 

. Evaluation is needed when designing a course for the 
simple reason $hat practically nobody is capalDle of 
doing. anything perfectly right for the first time, let 
alone something as complex as a training programme. . 

Moreover, immediate evaluation of the first part of 
tha^Mesign may save a lot of unnecessary work during 
the design of .later parts. One learns from mistakes^ 
and shortcomings. 

86 
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Inst faction ; 



Constantly called for when giving- 



keep ir^formed about the students^ progress 
draw tYic atte ■tion t. 'less fortunate actions of 



cak-' corrective measures 



- to check v:^^6thervt)]:ife objectives are still. 'being 
fc attained.. i^,^ ' 

' Eval^^^^^" is ^iso^neces.^fey to keepVthe quality of a 
traii^^^S Pi'^ogi^^mmt: up'^'to' date ■: ^ - 

- . te^chn^^xogy changes With * time, ^ s'6 should the con- 

tents or ^]ae courses " ' y 

QYi^an^es students' 'previous training cause va- " 
;j-iations their entry levels 

X^Yie stud^^ts' mentality changes, as does their 
ability, sC'.^ial background, etc, 

t^bc ment^]^ity and other qualities of the teachers 
^^f^ange- ■ 

■jn gene^^-^ v;e may say that both during the design 
a course a^^^... during the repeated running of a course 
^^^luat^^^^i is essential to optimize and preserve the 
^Uaiity ^^^t course. 



-^hOpjn;! EVALUATION ' ^ ' 

I'lnAhe o^^e of product evaluation v/e examine in how 
'j^^.^'^tli^ plann^^ objectives have been attained. These 

'h3:ve been specified' for the course "as a whole or 
" ^r. a-^cri.es lessons or even for one particular les- 



^'odu^'^ . ^^^luation is possible if and only if the ge- 
!^'r'al ^^^y ^he intermediate goals and the specific 
^bjeat:i^^'' h^v^ been laid down unambigu^tt§\ly and ' 
^ys.tematicalXy. ^ 



^^'TKODS of product evaluation 




are 



tests. These are tests for the partici- 
y^carr"be divided into the following types 
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tten 



oral hearing of the lesson 

talk or lecture prepared and delivered 

by the j^^ude^nt. 
gi'pup lecti^e", by two or more students 

- essay 

- 3hort answer type 



open questions 
c-losed questioi7s' 



ranking 

sorting v 
true/false questi?ons 
multiple choice if'' 



Auto-descriptive technjLques . The students are aske-d 
to givq their., opinion about . the objectives through : 

; - ^nq^iiries 

interviews of single students 

of groups of students (tv;c or more) 

3. Observation techniques . These are applied by obser-. 
vers. They may use : ' 

- checklists 

- rating scales. . ^ 

REQUIREMENTS that go6d achievement tests must fulfil are : 

The students must be told well in advance what will 
be expected of them during the test. |^ 

- - They 'must know the minip:ium achievajpent level that^J.s 
still regarded, suffici'eht . ^ 

^ - The' tests should ' asv'mlich as possible , cover the entire 
subject mattep -and not be limited to a part. 

- Very- good st udents./'Should be able to make all tests 
v^ell' in" the required time. 

.-^-'llhe scoring must be as objective as possible, that 
' is. Independent of the person who does the scoring. 

: ^ Of great ojiiportance , ^especially for product' evalua- 
tion, would be the following : To obtain information from 
EX-STUDENTS (i.e. from students several years after, 
having finished their training) on the question in how 
far the subject matter learnt has proved useful in actual 
practice. This canine doEe. by means of enquir^iee or 
interviews . ' "' * , •* ' ' < ■ ,. 



lS^r'i:;iMgly enough this is har^cily ever done. The 
ion^cari be attributed tq a variety of reasons : 

- It is difficult to ..iistinguish wha^. one has learnt 
within the. context of the course from that outside 

• that context. 

- Subsequent courses obscure the learning. effects of 
previous coui'ses.;' 

- Subject matter, ol-"' learning contend that is appa- 
' rently superf Itious may indirectly have a strong 

positive influence • (e.g. higher maths that the stu- 
dent, does no longer use may contribute to a better 
overall understanding or help him read the litera- 
ture on his subject 

■•- Many ex-students finish up in a different discipline 
through a change of function or position. 

~ Many ex-students are untraceable . 

PROCESS EVALUATION ' 

On the case of process evaluation the learning- 
teaching proci^ss is considered in all its properties, 
such as methods, contents/ materials, motivation. Job 
orientation, etc.. / 

Product and process evaluation often go toge ther 
and are inter^e'lated . 

Process evaJ^uation takes pl^ce ctntinually during 
t?h^-. design and i'liiplementatipn of a course. By far the 
.ifl^t way of "doing* this is having an experimental group 
%r ''.'participants go through the learning process while 
actuair^i designing the ccTurse. The first lessons are 
immediately' given a trial run and thus provide useful 
^^^^^^"^'^^Pj^^o^ -^^^^^ lesson^ so that mch unnece§s,^;ry 

' "''*^^'TavoidC?d . . .. 



'■'"^J^^'^'^'^^ '^^^^^S programme has reached^a>.stable 
~ ^^'^-^"-^^nual /evaluation is required during subsequent 
s'^oy^li-rse . This to ensure especially : ' 

„nat the^%"tyn&ag situation remains, well adapted 
■to the student^'-. /entry knowledge 

that the \ari(^s '^ei.'ements of the learning situation 
.are -mutually balancea',' -.as for instance a -good co- 
ordination of the diff^e^t lessons on one subject 
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that the learning WWreffljf^re'a'ds to the desired 
effects in an ef f it5i^^*>/ay • 

that the learning effects' do not gradually diminish 

that the objectives, contents and methods do not 
lag behind modern developments 

that the lessons are job-oriented, t^at is remain 
directed towards "practice in real life after the 



T^ie METHODS used in process evaluation are largely 
the same as those used in product evaluation. There are' 
descriptive and observing techniques. Achievement 
^techniques however ar^ of no interest here, 

1 , De^riptive techniques 

- inquiries for various categories of^ people, such as 
the students, teachers, specialists, educational 
scientists, people from the students' working sur- 
roundings, etc, 

- interviews, for the same categories 

- analyses of the learning content to be criticized 
by specialists on the' subject. 




2. ObggTvation techniques 9 

To be carried out by observers during the lessons 
or at the time that the students are to apply what ^ey 
have learnt -.t.o p'ractice, ' -^^ . 

■ " ' ■ -..^ . A 

EVALUATION CRITERIA ^^v- * ' M - 

' '■ ' " . '^j^. ■'^ 

Evaluation if it is to be dotae 'properly, must. me§t 
certa^ih requirements. These may ''eoricern the -quality of , 

the investigation or the method of application-. ^ ■ 

. .' 

A - The quality of '\^e. investigation 

1) Validity-- ^^ (=, the extent to which an ins trument rrfea-^J 
• surB^ wh'at it is, supposed to measure). ^ 
An- eval'uatiion must be valid. This, means that it 
fn'ust:,be ima*mbiguously .clear to bothVtlie inquirer^ 
, and' the respondent what the evaluation questions ._ 
mean, - Such ques tions.-:,rnay contain ' ^ . 

- stating a fact or facts " . 



^» ' ATT 
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- stating a relation between 'cause and , 
crt'ect 

2) Reliabil ity indil'ference to circumstances) 

The re liability oV an evaluation . implies that the 
re'sult ir. not influenced by the circumstances in 
v/hich it is aonc. These circumstances -may -be' rela-" 
ted to Person , Place and Time . 
Examples T"? • 

. - V 

- The £t\idents in group A must'^no't- be;- evaluated 
'in a dirfferent way from those in group B. 
Teach(^r X must not judgc^^his students in a; 
different way from teacher Y. 

X ■ V 

- The evaluation in Amsterdam must nbt be done 
in a way different from .that in Eindhoven. ^ 



- In . aui^umn as well as in spring, even years 
'la t.i^i^i^e evaluation should take place in an 
i d e n tt i 0 a 1 w a y . j 

- The application ' / ^ ^ . 

1 . Relevance (;= usefulness ) i , - % 

The findings must not Just be "nice to know", but 
"^.'jhey must be applicable to practice. 

^. Timeliness (ready and suitable for use) ^ * - 

The findings must not come too late, or in a form" 
that makes it impossible for them to be translated 
into. action . • 

3 . Communication (of f indirS^s to all concerned ) 



For instance, the results evaluat.ing a tiourse 
while in progress' should reStsch the organizer ^of the 
course in time as well as all other^'.concerned, such 
as ' the teachers , assistants , administrators, -etc . 

Support ^ \ 

Evaluation should ha\s the full agreement and s'up- 
port of the directors as well, asf ^11 trhoid concerned. 
If not, it may have , the wrong efjfects and especially 
the reliability will suffer. * 
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5 . Efficiency 



It_is obvious that the cost-benef i t relation i^ 
important for evaluation too : for instance, it 
may be too expensive to obtain resi^lts in time. 
Moreover, it Im Important to ^consider beforehand 
hov; to organize the evaluatiori to save unnecessary 
HP work : try to reach an understanding with the 
1 people concerned on an efficient planning of the 
qval'uati on. 

DOCUMENTATION AND EVALUATION OF EXTERNAL COURSES 



So far we discussed evaluation in general. As a 
whole it is applicable to training programmes organized 
at home. 



Vie shall now consider the evaluation of external 
training programmes . They ar'e not as easily influenced 
as those at home. 



.It is recommended to set up a DOCUMENTATION and 
EVALUATION of external courses visited by empl6y\^.rs 
from different firms and different branches. 



The DOCUMENTATION is needed for a'^Iilrvey' of all 
training facilities. This is a useful. tool for training 
officers who have to giv6 quidance. 

The documentation provides "organizational data" . 
These can be collected on one card per course (both side, 
are used) in one central place. For an example which ha. 
proved practical after many years of teaching experience 
see pages 9^ and 95. ■'i--- . ' V 

It is of the utmost importance"* that docujTientation 

is standardized. At the moment there is a jumble of 
different information leaflets and prospect^s'es which 

only too often - each in its own way - try lyo- camouflage 

shortcomings. It cannot be L;rged too much that all 

institutes present their data in a standardized way in- 
a standardize (^sequence . 

The EVALUATION on the one hand allows/ for a selec- 
tion qf the best and mo^t suitable course^^; This too is 
an important help in giving quidance. 

* . 02 



Secondly the evaluation data may provide' the 
necessary Liuggestions for improving e/.isting courses ^ • 
that' appear to be lacking on many points. Like the 
documentation data these evaluation data can be recorded 
on one single card per course in one central place. 
For an example that has also proved practical, see 
pages 96 and 97. 

The^ evaluation data are collected by means of 
INQUIRY FORMS . The questions on these forms ave arran- 
ged and worded according to the data required on the 
data sheet. The inquiry forms are intended for the 
following respondents : 

1 . NON- participants 

2 . Participants (students ) 

The forms for category 1 are only meant for a small 
number of people . . 

The :oi'iri:^ f-or category 2 are meant for large numbers of 
:-tudents. It is therefore necessary that they 
can be processed by computer, ^ 

The observations of non-participants and par.tici- ' 
pants together make for a picture of a course which is 
as complete as possible. 



COURSE DOCUMENTATION DATA 



date 



institute/ course/ code 



-Title of course or 
conf e rence 

- Institute/School 



- Aims of the .c'ouFse /• • 
or carif erejic!^ , 

1. Previous tra^r^li^g ,^ ; 
• required '^'-a^*; 

2 . Categories of people 
tile course is,, in ten- 
ded far ^ 
Hbiration of course 
or conference" 



3. 
4. 



Starting and ending 
date ( s ) of course 



Costs per partici- 
pant per course or 
conference 



6. Placets) and 

Address(es) where 
lessons or meetings 
are held ' i 



7. Curriculum survey, 
showing subjects and 
teaching. methods ^ 
applied 



Name 

Addres s, " 
-TelepI^iQTi'e 



meals and accnnmoda tion ; 
fees 

materials 
examination > 



total 



J 




95 



Name course or* 
conrerehce 



Code 



8. Number.. of partici- 
pants per class or 
group 

9. Number of lessons 
,or sessions a v/eek 
(a month) 



10. Total numbers of 
lessons or ses- 
sions per type 



1 1 . Gi^dance or coaching 
or participants 



12. Skill training on 
the job 



15. Method anci frequen- 
cy of- testing 
acquired knowledge, 
■ skill, etc. 

14 /Examination date (s ) 
place{s) 

norms 

15 .Materials composed' 
or revt'i^^^d for the 
. last' time ■■ 



; \ Minimally : 


Maximally : 


... written a . . ... 
... oral X a . . * ... 
. . . workshop- a . ... 


. . . Written 

. . . oral . 

. . . workshop 

y ' 




Yes/No ; foUn oi" 
coaching : 


quidance or 


practice on the 
day supposed to 
yes/«o ; on what 


job during the 
be present : 
terms : " 




• 




IfT't^ - - - 


part . . . 


in year 


pai"t 


in year 


Dart 


in Vf?ar 
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COURSE EVALUATION - DATA 



Mate 



i n s t i t u t e /cjD ur s e /c aye 



Title of course or 
conference 

I ns 1 1 1 u t:e|^Sclip*6 y 







Name : 




Address : 




Telephone : 







quality of inquiry : 



'3 



Aims bpl^E-e. o 
confertencej^"' 

, Inquiry irttb sui- ^ethod and done by 'whom 
tability of parti- 
cipant 

, Previous training 
required as eviden- 
ced by former par- 
ticipants 

, Information about 
.content anci aims 
of courses of con- 
. ference ' '''>y^ 

, Examination 
requirements 



. (too much 
prescribed previous, f sufficient 

training , . (insuf f ici.ent 
better alternative' (if rte'cess^r.^^) : 



gives QOpi'/ sufficient / bad ih 
si gilt 



^5 
6 



Value of diploma 
Possibility, of re-' 




rtit what* the course offers 
..cterined. we^ll/suf i'^i-cient 

^ gher 

qual level than supposed 
1 owe r^>'^ * • 



applying the efforts gd;^is^O,:l:ess difficult course 
.made if the course ! ^^wf 
appears too heavy >".:wy^^' 
7. Investing risk, 
regarding 




^ t^nS^^^ ^^^^^"""s^all/mediuin/large 

itiSration of , > ^ . /, 

^ course : small/medium/large 

^costs : small/medium/large 

job orientedness small/medium/large 



, Ay.erage number of 
study hours outside 
course and working • 
time ' . I 
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ERIC 



Name courije or 
corvf erence 



code 



9. For the eval,*uation 
the following peo- 
ple have been 
cdnsulted 

10. Level of : content 

pa^^t icipants 



11. Content is in 
t general 

2 '.Content was last 
modernized 



13. Evaluation .of ■ 
courge content 



14. Evaluation of ^ 

learnin-g' situation 



teachers 

Instructors methods 
applied 

student guidance ^ 

accommodation 

Evaluation of - 
course organi - 
sation 

16. Evaluation of job .. 
" orientedness 



participant^W'instructors/ specia- 
lists / educatiorial scientists' / 
industry 



low / intermediate / advanced 
low / .intermedia\:e / advanced 



progressive / up to date / 
behind the 'times 



subject in yea.r 

subject in 

subject in year 

subject ..... in year 

subject in 

subject in year 



EVALUATION BY NON-PARTICIPANTS 



The category of persons for whom these forms are • 
intended include :instructor staff, experts (.on the^sub- 
jects ^ taught )', educational scientists ancl "industry . 
To 'spe<?ify :/ , . ' ^ ' 

• - ' ' / ' ' • . - r . 

- instri.ctor/staf f are those who are required- to give. 
instrucDion in a way prescribed by others and with 
'means p)?escpibed by others . , , , 

^Th^e expert envisaged here is- a person wej.1 versed in. 
'the subject treated in the course. ^ 

-An educational scientist is som^ne whose, expertise 
• li^s in the field of educational .technology^ • pedagogy, 
, learning psychology, etc. ■ 

- "Industry" here is meant teT imply : industrial adminis- 
trk^lon ; the training ofAdcer ; the personnel manager; 

^theparticipants ^ superiors or colle|^gues. ■ 

N "One may wonder which of these categories can be 
approached in pract^icd ? The following example^ may be 
Helpful to illustrate the point.^ / ^ 

- As-inctf'urbor one may approach a person in the company 
who lectures in the course. Or a tutor who c'oaches 
individual students in case of correspondence courses. 

- An expert could be a tutor or coach mentioned in the 
previous paragraph. Or he may be a specialist in the 
company who is much concerned with thd thorough forma- 

'tion of the people forking' in his department. Again 
he mav#be an examiner, number of a bo^'rd of examiners 
OP appointed observer. " ' 

- Morfe difiyi?^ult to find is the educational sciejitist ^. 
■ It shoufd be possible to find someone in. the ceH^ral 

training department who is willing to evaluate a course' 



An- industrial evaluation form should be filled out by ^ 
.pne person if possible. This-^may be the .training of- ^ 
ficer, the personnel manager, or the -participants ^ de- ' 
'partmental head-. The idea 'is that the person who fills 
iii the inquiry consults the- other ftategc5ries mentioned . 
in order to come to an 'evaluation -that . can be<sjBaid to 
be ^ representative for that fir^m. 
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ir not all of these • respondents can be found,, t^is 
is no serious handioap : ; the specialist's opinion'.by 
Itself e.g. may be quite' sufficient to come to^ a general 
conclusion. ' • ^ 

EVALUATION BY . PARTICIPANTS 

Th'fe .training officer receives the forms in dupli- 
cate.. Both of these forms he hands out to 'the partici,,- 
pant -together with a copy of "explanation for the 
participant", with -the request to fill in- the forms with 
care and send them "ba-ck to him in a week. • ^ 

"- The trainings-officer may keep the YELLOW forms for 
his own administration. The WHITE forms are to be sent 
to the processing 'centre . 

Some thr^e months after the finishing date oP the 
course the results will be sent to the training officers. 
They show : trte degree of precision of the "objectives, 
th^ pedagogical merits or demerits organisation, acco- 
modation, jc^b orientedness, learning effect. 

^ This makes it possrble to compare different courses 
or to 'compare the course with. another one of average 
quality. 

APPLICATION FOR GUIDANCE 

The forms for participants are also suitable as a 

means for guiding or coaching intlividual students while 

they are taking a course.' Guidance is . of special impor- 
tance in case of long training periods. 

If, for example, -a training programme lasts_ one* or 
more years, the Sorms may be used at intervals o'f, say, 
10 weeks informed abnut progress made. In these cases' 
you may keep the .foJ^ms and interpret the dat a yourself . 
If, however a substantial part of the course is completed, 
e.g. one^term or year. It is advisable to send the forms 
to the processing centre 

' The evaluation data on tk^t part, of the course 
'will then be sent' to you. * ' 
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(d) THE STANDARDIZATION OF .THE TRAINING 



OF MASTER CRAFTSMEN IN GERMANY 



"Industrial Master Craftsmen" have been trained in 
Germanyfor several/ decades . ^ The training form descri- 
bed below concerns the further training, of fully trai- 
ned skilled workers by means of special courses, which 
are provided by various bodies and are very varied in 
content. It^^isvjDOssible to take the "industrial Master 
•Craftsmen" examination through one of the'SO or so 
chambers of industry' or commerce/ This examination is 
generally -reodgnized throughout industry although i,t is 
not sijtije,ct to state control, 

Significance » * 

Examinations for the qualification of industrial 
master craftsman were originally' relatively insignifi-* 
cant since most master c-^aftsman examinations were 
taken in trades skills (about 52,Q00 per year) and the 
majority of these master craftsmen were working in 
industry. In course of time the industrial master 
craftsman examination has assumed increasing importance 
because; it ^is orientated towards the specific problems 
arising from employment in industry and because the num- 
ber of master craftsmen in industry, who had taken this 
exam had become, considerable. Many undertakings-, parti- 
cularly the larger ones; have for. a while required of 
candidates for a post as master craftsman tha^.they* 
should either have already passed -the industrial master 
craftsman* s* exam or that they should take it without 
delay. In. addition many companies train industrial mas- 
ter craftsmen themselves or finance their traiping. 
Since ^1969 participants in such training courses can , 
also obtain grants Tor subsistence, instruction, tools 
and traNLning materials through the Federal Labour Office 
and considerable use is made of this possibili^, 

SJbatementy of problems : 

1 , Problem / No uniform arrangements : 
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IrKiu^.ar'la.1. ina:jter craftsman training is organized 
ndther'^by the State nor 'by private industry. Only the 
German Industry and Ti'ade' Counci 1 (DIHT) - the central 
association of all chambers of commei'ce and industry in 
the German Federal Republic - published in 1956 an out- 
line syllabus as a recommendation which is, however, 
only adhered to by some of the .bodies organizing trai- 

*ning. It^\is not^ijoi^sible to speak of a uniform pattern 
of training. The final examinations set by chambers of 
fndustr-y and commer^ce are.ialso similar only in certain 
basic features. Similarities, however, are only the ^ 
result of the ^subject being examined, rat^her than as a 

.result of regulations or ag^ements. 

Since I969, on the basis of the Law on Vocational 
Training (BBiG) the Federal Minister. of Education and 
Science ii^ empowered to fix by decree uniform arrange- 
ments throughout the. Federal Repub^c; concerning the 

- content of. training courses, 

- aims-**; X,,,,=.=i=-«i-==''^^^ 

- examination- requirements, • ^ 

- examination procedures, 

- admission requirements and the, title of the final 
qualification for the industrial master craftsman 
examination. ' 

As a result since 1973 the Federal institute for 
Vocational Training Research has been carrying out preli- 
minary work: on behalf of the Federa'I Ministry of Educa- 
tion' and Science in preparation for the publishing of 
'these examination regulations. 

.. Proble-m : Diffe rent irlterpretations of the, concept 
of industrial master craftsman : 

It is difficult to carry out preliminary work for 
these examination regulations sinoe in the German Federal 
Republic a master craftsman' in industry can be engaged in 
numerous activities which differ in specialist content. 
If they were asked it is a Isg^ 'certain that even more 
brandies "Of indus^try would report the need for special 
training for industrial master craftsmen. 



In addition to this widespread differentiation as 
re.gards specialization differentiation as regards func- 
.ti'on has developed in the acfe-ivLti'^ of the industrial 
master craftsman : in small and medium-sized^^^undert^ings 
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master crartsrnen have more comprehens^^ve . functions akin 
to those of managers-, whi^h may even extend to the fol- - 
lowing areas : •. - ' ■• •> \, ' 

- recruiting, paying and dismissing staff , 
supervising out manuf acturlrng -and ^product ion work , 

- calculating cost and material requirements , 

- purchases, sales and other dealings with ■ oustomers . 

On the other hanti ind^strial master craftsmen in . 
large or thoroughly rationalised undertakings are. no 
longer concerned with this side of activities, since 
they have had to ce(i^ many of these tasks to others. 
Normally .the above-mentioned work is carried out by the 
personnel department, the planning department, the pro- 
.duction department, ■ the preTiminary costing and ex-post- 
facto costing departments, the purchasing department or 
the customer service department and other departments. 
In these departments there are master craftsmen ^p'^ciali- 
zing in the specific related tasks'. 

As a result the.r^ is talk of a devaluation of the 
functions- of the master crafi:sman. However, it should 
not be overlooked that even^in highly organp-zed underta- 
kings there are still "industrial master craftsmen who 
continue to fulfill an important function,' It is clear 
that this lowest level of management cannot be dispensed 
with. Thus the question here is what functions the^ 
industrial master craftsman retains and which have been ^' 
added to his oHsi^^l functions. 

It' is of course necessary to proviae .answers to 
these questions before they can provide the basis for the 
development of ^^vy, uniform e:x!amiriation regulations 
throughout the Federal Republic. 

The complexity and complication or the problems, 
with which we are faced will have become apparent. 
Unfortunately the problems are, at least in appearan'ce, 
complicated even further^ as .a result of the fact that 
we are required, before taking any step, to seen the 
advice of an expert joint commi^tteez-fljaSe' 'up of represen- 
tatives of the employers' opganizayions and the trade ■ 
unions and which also includes * tes(chers from vocational 
training schools. Thus it is eas^^ to see -that when^these 
questions ape being decided we are not concerned alone 
with questions of "specialized content and functions, but 

102 ' " 
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that very i^tfong' economic and proTessional interests 
are involved. • ' • ' 

3. Solutions * ^ 

' First Stage ^ ^ ^ 

'"In vie^ of 'the range of problems described- the 
uniform arrangements for the training of industrial 
master craftsmen should in our opinion proceed step ^ 
-by step.-' This means that the issue of a State order 
must hev preceded .by a fairly, long stage of several years 
during 'variou^^odel- methods of * t:.raining industrial 

mas^r c^ftsmen are tried out in different trades and 
at differerrt; functional levels on the basis- of recommen- 
dations and the corresponding examination requirements 
f?iaould .."be .orietiJ:eri to suit .p_ractical and social require- 
m'ents . ' » " ' m 

''In this process compromise would .be unavoidable, 
which might possibly^ ri/n counter to our scientific 
principles . \ , 



Sey^nd Stage 



. Division of industrial master craftsmen according 
, to.fieW.of activity must be kept to a minimum. The 
drawing up of arrangements for variation in courses 
content according to the subject must' be left to specia^l 
expert '.eommit'tees from the trade associations. 

T'he Institilte for Vocational Training, has to concern 
itself ^>rimar:ily with th,e. aspects of training and exami- 
nation "Of master craftsmen which transcend the require- 
-ment^of .^individua 1 fields. 

We -consider for ins.tance that the following quali- 
fications can^be required of a modern, industrial master 
craftsman in addition to his specialized qualification : 

- organizational ^bility"'"^'"^^ 

- ability to communicate " , .. ^ 

< 

- ability to impart information ■ , 

- ability to cooperate ; * ' 

- adaptability ° ... 

- ■ ■ ■ . i . 
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- capacity Tors. reu<kM'::;hip. 

It i3 clear from this^list that thase abilities in 
part depend' upon eaeh other. At present the aim of our- ' ^ 
work 'is to express these abilities in ter^ms of the cur- 
riculum. Added to these qualifications which transcend • 
specialized knowledge is specialized knowledge in the 
•individual subjects. Wher;efe t!us latter knowledge canV 
generally T^e impartjCd bymeans of courses and. testedin \ 
examinations without great dil'ficulty^ the problems of \ 
imparting the qualifications in the above list by means 
of courses^ are very great. . ' ' . 

Third IStage • ' " 

'i ' • • 

The differentiation as regards specialised content \ 
and between various levels of function is^baied upon a 
pattern of industrial ■ master craftsmen which is best 
shown in the form* of a"matrix : 

.1 -I V ' 

- the matrix could ' perhaps take the .I>orm''given in Table. 

This.'matrix can be read and evasLuated as follows : 

'■' ..] ■ ■ ■ ■ ■ 

\r we refer tc3 the sector entries (vertical) then 
a training which comes under one o:^' these headings ful- 
fills alP the 'functions represented would produce a type 
of master craftsman who could be described as a secto- 
rial master craftsman. He can expected, to be.ab^e to 
deal- with any problems which are likely to face a master 
craftsrnan in this, sector. 

. This would mean that a master . cr.aftsmari so trained 
would be able to demonstrate his full command of- planning, 
^prociuction, .madhine installa tiofi and tooling-up, direc- 
tion of personnel, ^nd dealing v/ith customers in this 
sector. 

is often maintained that the present-day master 
craftsffian fulfills these requirements. But if his -.trai- 
ning is examined then we di^coVer that it is primarily 
oriented, towards prod uo1ifi.on'^^t asks, knowledge ^f machi*- 
ne'ry and the training of "applrentices. Questions such as 
organization and »planni^rg;:techniq^ies,^; direction of per- 
sonnel and problems of cooperation with customers, supe- 
riors and colleagues occupy, relatively, a very subor- 
dinate place. * ^ ^ 
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II' wu wxarriijiu .t:htJ runction enti'ioL' (horizontal-) in 
the rnatr-ix ami if.' we, imagine • tfh'.^t tlic training I'ollows 
the pattern pr^.^:;.ented in thcGe^errtr ie^j then we could 
, assume an inclu.strial inai^tor' cral'ti:;!nan who would not be , 
' rnoliriiy coneerfie .1 with the product pt^oduced by the sector 
butf' wj-th tl-ie-wor'k that ha^ to be done before the product 
c^n be pe'oduced. Thuc> it would be possible to deduce' 
kinds of m?ister 'cr':U'ti::nen concerned with certain func- 
tions, e^g. a ma.^,tor' c ra f t^^man • in' planning, in praduc- , 
tionT in ma^chine ry , in supplies, and in customer service 
etc,. I "such -d view of the industrial master craftsman is 
.of tbri rejected by those who are responsible for the 
training of' iTiaste.3|^crai'tsmen. They regard this as 
^turning master .crSrtsmen into technicians. However, if 
the career pattern of the average industrial master . 
craftsmen is examined, ' then one" notices that although 
master craftsmen who have specialized in a particular 
function in one sector often change sectprs they equally 
often kee"p to. the same functi6n. Obviously .then there 
is a potential of greater- mob; 1 ity . ' As regards -the^,^ 
future this development is further encouraged .by- the 
fact thtit raw material' is being processed less gnd less 
by men, i.e. specialist workers, directly, but rather^ 
by machines. Ih the Federal Republic this trend has 
already mad-e itself- very clearly apparent in the training 
of apprentices and has ^necessl^tated -considerable changes. 
The basis^O-f the argument is. that in the ca:se of indus- 
trial master craftsmen, too, ,there will be a fairly.- 
strong, tendency' to develop into functional -master crafts- 
men, although the problem is hot definitively clear. 
Unfortunately the question cannot be answered directly 
by means of the system represented., above, since like 
any system it presents only a ver-y simplied picture of 
possibilities. 

Concluding remarks : , - ^ 

It is probable that the trer^d is moving in several 
directions and ^the result will be : 

^ sector'ial master cral tsmen ' 

- f unctiona 1 master craftsmen . ^ 

- a'hd'master craftsmen concerned partly with the sector 
and partly with functions. 

O.ur task, namely to^wor^out suitable' training pat- 
terns , where necessary wi th\s)3veral variations , and in 
due course, to test them out Vith the -help off suitable 
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promote fi-, ii; :.;U 1 1 , important . -°''pm,,;^ > -tfijl .exarril,,. We 
consider it po;::;ibi(: to develop ^' 3^ J^i-iy*^ , , ^ „at ^^i"^" 
requirements and to I'ormulate the" -l-T) ^^^jii-'y ^i'tigh^ 
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for the grouping of Industrial Master Craftsmen .by secior and 
function for purpQ_r,e,G of developing appropriate l^rthelj training p/ografnines 
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Ex^le Combination of . 

master craftsmen with par- 
tial functional responsibi- 
lity arid partial sector' 
responsibility. 



!,xample : Master 

craftsmen for all sectors 
of the metal working 
^industry. 
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UNEMPLOYED ^ADULT^ 



\ 



. Thi:3ir. a UKf; ove I'nniunt 
Servicers Coinrni:;:'ioti to' bo 
ning Servicer, Ageru;^^. 
courses for' unempIo:^cd 
:A:)ility to create, ^nd 
own job opportunifcie 
five pilQt-coursf?s\ ar\e 
unemployment. An 
the scheme from thev^Litset, 



initiative fr-om the Manpower' 
implemented through, the ,Trai- 
Thc basic idea'^ is to provide 
persons uo that th^y ^acquire, the 
r 0 s> pc n d e a 1 i s tl c a 1 1 y t o , their' 
• Thu sohenie is ^experimental and 
to be mounted in areas of' high 
uation plan is to be built into 



The Orricial Objegtiyes 



(As skills training is not given, 
are referi'ed to .as clients rather 



persons on the cour^ 
than as trainees). 



(a) 
(b) 
\ 

(c) 
(d) 



When the client leaves the course he should : 
be motivated to seek employment^ ; 

be .familiar with the work outlets in his area, and 
be realistic in his choice of jobs ; 

know how to seek, appl-^^ C^Jr, and hold a' job ; 

learn to use his initiative to asse:S.s.the needs of 
a job, and learn to perform it successfully. 



Related toythese offici^il objectives are the less 
tangible aims which are embraced in the notion of social, 
education. This involves acquisition of the sociabil'ity 
and comportment skills which are necessary for. coping 
with life generally. Social skills are required for 
interacting efficiently and smoothly with other people 
•both outside and within the work situation ; e.g. tal- 
king to strangers, taking 'instruct ions from supervisors, . 
communicating' efficiently . By -contrast life skills do. 
not necessarily involve interc^trt ing With other people- 
but include .all those things that are necessary to co.n- 
ducting one's life ' ;. e.g. calculating a wage '^packet , 
home finance, getting -information/ making complaints. 




Died, with a more 



finding accommcfdat Ixon,^ moving to a "-new areai^, j 

*Th^ general: objective' is to enable the 'indrfviduaj.^ ^ 
tq find, secure" andri^tain full-time employment, and *at 
the'.same time nuake realisti*c career, decisions based^^on 

Lnowle'^ge ratlj^er than guess-work ,r rcoupl 

raequafe aS Ji^stment ^ to l£fe in-/g^eral.' 

Target PoipUlajDion \ , • ' ; ^ 

• Thb : unemployed ,and-freguent. jpb-changers , .gpneVally 
at - the sub'^apprenti-ce l^vel, but excluding "^he pl^ys^^ally 

• anc^ine'ntaI.ly hand i Gapped.' The c>ients fall into two' 
groups ' : - ".^ 




(a) unemployed school-leavers, boys and'.-gi^ls , 
^ ■' aged 16-18 ; ^ , ' ' ^ 

-(b) twnemployed adu*lts, -aged L9+, men^nd women. 

\ T^ey are unemployed ^ for a great variety of reasons. 
'Kiese include problems of social adjustment, 'a lack of 
interest in work,', lack of confidence, a^ negative atti- 
tude *to themselves and society, ignorance of the know- 
how for getting jobs, th^ effects of traditional or 
intended unemployment, or general social and educational 
djfeadvant'age^s I^ahy of the clients will also have 
criminal recprds. The range of ability will vary "consi- 
derably ^though most will be of low general abi-lity. 

Clients are recruited for the course by the Youth 
Employment Service (or Careers Service.) -for young people 
and tne E.S.^. for adults. 

Duration of Courses 'd ^ 

In each c?ourse there will be a maximum of twenty 
clients with three full-time instructors. There is no 
block entry. For adults the maximum djaration of the 
course "is twelve weeks, and for young persons it Is 
twenty-?bur weeks. Clients are, howevejp, encouraged to 
leave the course as soon as they feel'^^ey are ready. 
There is a small financial incentive to attend the 
course as the training ^allowances are in .excess of 
unemployment benefits. ; 

Job PlQ^c^ment . . ' . ' 

It is* n\>t the aim of the course J}0 place each indi- 
vidual in a Job. The idea is to develop the ability and 




initiative to do this fpr himself on leaving, the course 
anTd in the. future, 

Training of the ^Instructors 

. . # ■ / 

Instructors for the Wider Opportunities Course were 
chosej^f^om existing SkilJ^c^ntre instructors according 
to specif ications ' which dealt -with both their craft 
ability, and their temperament and' ^personality . Instruc- 
tors were not chosen according to trad,e or cpaft, l^equi- 
rement-s but' rather foi^ their overall sui.tability for 
'the Job^ Training consisted of- a three-week- course : 
two wAKs /of .class7room ins^ructio^n^ discussion and ^ 
pract^tl/ work separated by on^ week of 'practical pr^o-^ 
Jjgcf^rk.' The maij;^ topics included understanding the 
objectives,^ the use- and design of discovery learning, 
the use of a leaning taxonomy, .the no,tion odn^arning- 
to-learn course, content material, discussion learning, 
, motivation', ^.nterpersonal dynamics,-' social si<:ills, and • 
' techniques .for assessing ana -guiding the individual, 
T]ie general aim was to eq.uip the instrucrt;prs for the 
especiat nature of th^ W?0. course and their special , 
(perhaps unconventional) role in it. To this end' the 
course was designed to give them some of the skills and 
insights normally possessed by "professionals" ^.g, 
psychologists, training designers, social -workers , \^ 

Nature of the Course 



(a) The section - 

Ideally this should be located under one -roof, 
preferably in one large work-shop. The section typically 
contains a representartive selection of widely used in- 
dustrial machinery e.g. centre lathes, capstans, milling- 
machines, drills, power saws and presses. There should 
also be metal and wood-working benches, welding-bays, - 
drawing-boards, desks and office equipment -such as, type- 
writers and photo-copiers. In addition there should be 
opportunities fo;:' painting, decorating, brick-work aRd 
horticultural work. There should also be -rooms off-set 
for written wprk, office simulations, discussion groups, 
a^d self -administered -instruction packages. 

(b) Ro le of instructor 

Because no formal training is given, the instructor 
has an unorthodox roleV His main function is to guide 
and counsel the client."" On the basis of assessments made 
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by the instr'uct9r ^see 'Skill Samples below) he helps the 
client choose from the many alternatives available tvO 
hirn the course., -The Instructor ™ encounaged to 
aevelop.a vv^rm, olose relationship with the client, 
takij^ care to .avoid the traditional rormaT relationship 
with its' emphasis on ^he instraj::tor ^ s. position of autho-, 
rity over the trainee. Discipline is maintained as far 
as possible though trust and mutual respect. Conse*- >/ 
quentlyni^a flexible attitude toward lateness and poor * 
attendance is adc>pteG,as it is assumed that for these 
cLients, at I'east, punctual ity 'and attendance ^ are pro^^ 
blems they need to resolve. Clients ca?n be Jdismissed< as 
a last resort, but only if their ^ofetevioiir is^upset^ng 
or disrupti;jig the other clients. 

• . . " /// . 



) Course Cc5ritent ^ ^ , - ^ - 

* There is no set* syllabus ♦ The; list of equipment 
abov^ giVe^nPome indica^^on of the range of* activities 
that are 'available to the client. The ■ act ivities in- 
clude bench' work and metal-working^ clothing .^repair, 
gardening, electrical work, sewing, electronic assembly, 
clerical work, working in concrete, pipe- fitting, 
photography, sailing, calculator operation, h^J^ital ^ - 
work, deliv^e^ Jobs, cleaning jobs, hotel'and catering. 
This^iList gives some idea of -^he comprehensive range of 
skills, job types, a^jid job locations which the client 
can experience. rlHe can choose on the^basis of both his 
interest, and his | ability^.o learn the/basic skills in- 
volved. He can Spend up^O^kree or'four weeks in 
one area befone making a "further ch6ice to try others. 
On the other hand, he anight give,, up one^area after a 
few days and try some^hin^g else. / 



Cd) Methods 

There is considerable empTiasis on person- cent red 
learning, with cofis ioierable opportunity for individual 
instruction at the learner ^ own pace. Wherever it is 
ap'^opriate discovery learning methods are used, and 
there is also extensive .use of self-administered program- 
med texfs and project kits. 

On entry to a courTe^^^^He client discusses his work 
experience,- interests and aspirations with the instructor. 
As a resullt of this discussion the instructor will admi- 
nister -a small number .oX skill samples. These are^pe- 
riods of learning a representative sample of a skill. ■ * 
The client is first taught the task (e.g. a hal/-lap T 
joint) and is then asked to^ perform the task without. help 
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from the instr^uctor vwho. then rates the clients ability to 
learn the skill against a standardised error check-list . 
Using th'^"* results as an aid the instructor can advise the 
client in his choice of project work.' Equally the client 
can get a taste of the task to a job area^ without threat 
to his self -cocif idence . ^ ^ . 

Following the'^'Skill (samples, the client^^may c*hoose ^ 
to undertake a few Trial Frcrjects w)iich are simple tasks 
involving a coupfe of'^days work making an pbject, with 
only a small amount of instruction in b§^i c .techniques . 
If a client is' particularly interested in one project , 
he may go on to a fall project which . can take anything 
i^p 1;o four. weeks. -Havirig- completed this -he may try out ■ 
new skills(^n the form of Skill samples and Tria^l Pro-> ' 
jects , " / - ' ■ ■ 

• . ' ^ ' " 

Alternatively he may be sufficiently interested -in 
^or\a-area to undertake Job Rehe^rsal^, i^. one week 
periods working fbr/4n •employer *and at Wkf end^r^^ which 
he may returri^to^^j^^ course, or as sometl|ne,4 may^ happen, 
jDe tf^ken on permanently by tKe employerM Attitude . deve- 
lopment and social skills are taught by meaiT* «f-'-^discuS- ^ 
sion groups^ films, role-playing and .foiipial exercises . 

■ ' ^ . ' 1* " 

■ Considerable emphasis, is placed on learning the - 
skills of job-getting (e.g". looking around, telephoning,^ 
being interviewed). ThiS^^is considered to *'be a very 
important part of the cour^f^ 
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(r) RESEARCH METHODOLOGY ' AND TRAINING 



Educational and training research. ha\^^ to a large 
extent been carried out by analogy with thej -physical. ^ 
sciences. Evaluation makes use of such research 'and is 
i'tself aften a part of it. The properties and-Timita- 
.tions of research methodology are therefore crucial to 
the proves s%of evaluation. 

. Z' The typical experimental paradigm. is : . > 

' Pr'e-test ' . Process P.ost-test 

A / > 

Group 1 ^1 ^ • Method.i.A, X2 

^ ^\ ^ 

Group 2 Yi MethO(5^ B ■ ^ Y2 

The i^ssumptions pfi which the experimental^ method 
is based are that ind'iyrLdual test results may be ave- 
raged'; .that groups or^individuals with the same , scores 
may. be t?^eated as the same -that different methods may 
^be objectively defined and , manipulated. However, it is , 
"now often recognised that such assumptions are difficult 
,to realise in practice. ••'T^^ pre-test can ensure only 
'to a limited extent that the\two groups are comparable. 
Even individuals with identi-cal scores'. are almost certa^ii 
to be unequally equipped to learn the relevan^t ' material . 
Similiarly, the post-test will give only a partial mea- 
sure of actual learning and may be distorted in various, 
ways. 

An even more demanding task' is ensuring that the 
teaching processes differ only in the intended way. It 
is usually difficult to change 'a method of * teaching 
without at the same time changing the content in some . 
respect. Neither is the&^ a ..satisfactory way of con- 
trolling^ trie "teacher variable". Either the two methods 
are run by different teachers, in which case' the two 
variable^- "teacher" and "method" are confounded, or the 
same teacher must u^e both methods. In this case therfe 
will be an interaction effect between teacher style etc. 
and 'the method. It is well, established that- the .prefe- 
rences of a teacher have a potent effect on the. way he 



114- 



uses^a Ynethoci and the results it producer... It is thus , 

• dif f icui't, • if riot impossible, to experiment with 
"method"" as a pure variable. .And this leaves o^it of ' ' 
account the interactions between l^rner style and^ - 

■ \ method and between learner style anS teacher ^§,tyle. • 

In principle such 'limitations could be overcome by , • 
a combination of^ -exhaustive testing with sufficiently ^ 
sensitive tests^nd by the use of sufficiently large 
samples ^of learners. ^ Howeyer these re(^uirements\are 
very s'tringerit and far: -exceed the normal resource avai- 
lable' in terms ""of learners, money and appropriate tests. 
It is therefore inevitable that 'a -purely experimental 
approach to quesiiQ.n^:.of teaching and learning can pro- 
ducg only tentatS^^^<s;^.ults' ratheV than definitive con- ■ 
■■ c'lusions. ' f ^ / 

> Fundamentally, the. prottem is^th^ human learning 
Vannot be reduced to bits measurable behaviour. ^ / 
•.Behin(^ behaviour lies meaning, and Yc, is this that' is / 
)of central concern^ to- the teacher , /trainer and evalual^or. 
/Once this-^s recognised, the procefe's of evaluation* c^n ' _ 
' be seen -as^i^iterpretive and not a' matter simply oV / ^ 
objective measurement. Measures can be incorporate into 
the interpretation and. can act as a stimulus to f i^rther . 
attempt3 to understand training .events, ^ butf.ratirrgs or - 
test -scores by themselves can only b£ ambiguous/ 
'* ' ' ■ ' ^ . * ' ■ 

It has also to^.be accepted that interpretation is-' ■ 
itself a subjective matter. If it is .analytical arid 
makes vali'd'' use of data, a sufficient degree of . inter- 
. • sut^jectivity can be attained, but this does not fulfill. • - ■ 
' the notion of "objectivity" which has been' sought: from • 
the experimental paradigm. Neither is evaluation a ' . 
neutral process. --The act of observation is an active . 
causal factor wHich is always, likely to change th^ situa-/^ 
tion at which it is directed. Th-e presence of an evalua- 
tor, or the knowledge that an .evaluation will be rqade is - ■ 
frequently.j^-significant determiner' of trainers' actions. ■ 

Evaluation is therefore a social process. No amount' 
* of reliance on' "objective" t'echniqutes * can .change that, 
oply disguise it. ~ The eva.luatar's 'basic 'skill therefore 
is that of understanding a social process which Includes 
himself. He may use checklists, tests or idealised 

• dels 'of training but they j^an only' be 'successful 
aye used as' aids to understanding people's intent: 
implicit theories and . objectives . 
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CHAPTER 4 



THE' COST -^BENEFIT ANALYSIS QF^QCATIQNM. TRAINING 



One of 'the most di3cussed Issues at the Seminar 
\was the question of what significance should be attached, 
to^ Jthe results obtaine d by the application of cost- ^ 
benefit antelysis. to the. outcome of training programmes, 
yiscugsio/i w^s centered around two broad issues ^ 

(1) \what is the relationship between Cost' Behefi| Ana- 
• J)lysis (CBA)- and "social o^ethical" clecisio^ns ? 

(2) ^What are the particular techniques and problems 
- 'encountered 'in applying CBA to training ? . * 

1 - The scope of CBA 

In CBA we are concerned with tjje *^onomif* as a whole 
and not any smaller^ part of^ it s'^ufefi as the firm. Instead 
Qf asking whether 'the owners of the firm will become' 
better off by the firm enga'ging in one actiV/ity rather 
'than another the economis^/ asks whether S'ociety^as^ a-' 
whole will become betj:er /of f . by undertaking this project 
rather tha^ not unde^^klng it, o*r by undertaking instead 
any of a number of alt^^'native projects^ But' once .the 
net' is 'thrown wider and the -repercussions over' the eco- 
nomy at large ^'are brqiight ..into"" the calculus- a. number 
of problems afise which re'i^uire exten&eQ treatmen>t,. C^ief 
among these are the distinction between benefits a!nd 
transfer! payments , /the -concept of shadow pricing ^xteTnal 
economies and diseconomies. » A ^ 

. With soG^la^L variables th^* problem lies deeper. /C^n 
we, even 'in principle,' attaq^ a financial quantity to the 
social value 0f increased employment, f or 1 exampljev.? 'It 
4s in cases siich as this- that opponents of financial ana- 
lysis'.s^ee a/danger. Will not administrators feel more 
confident or objectives^ which are quantifiable in'eaj3h 
.ternfis ? "^here is perhgips then a danger that financial 
analysis .will pre-empt the-'wllple field of training policy. 



/ 
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^ that it will cease to be simply^^ way of Judging some of 
the possi*ble choices and will become the whole b,*is for- 



po 

choJ.ce 



\uch a digger is undoubtedly real, but so long as t 
traipPng cos^ money, some 'kind of financial- anal^is isl^ 
inevitable.*^ ^ 

•The choice is between forms of analysis wlr 
explicit and forms which are implicit and uAdec 
It\is therefore essential that the relation^ip^ 
fjlnancial and "humane" values be rationalii^ed, i 
ting each to be handled by complementary techniques\ 

\ ' ^ J 

The use of cost-beri^f i t analysis Wa^s attracted 
criticism for its tendency to pre-empt values which are , 
moral or political.. The fact that this can"" happen is a 
direct re^ult^af. muddle and lack of clarity about do-, 
mains of decisions. If actions within the field of 
training policy fd^ instance are made'and judged without 
reference to the wider domains, then the values appro- 
priate to those domains are naturally unrepresented. 
me fault lies, not ^in any form-of financial analysis but- 
Ln ^le impoverished context within^which it ^.s used. 




A property* of the technique whjtcli/ is not of terr ' 
recognised is .its contribution precisely to the process^-^iiinj 
of value judgjsment^ By afttempting to quantify Ini^}^^^?^'^ 
and 'output^ it fQcuses deba'te and cUri^'ies issue'Sj/g;^^^^^^^^ 
v'^ue in a parti:cularly acute way "^'Ijlh' the correc%'p^VlfJ^^^^ 
t^t therefore, , et^ljical values and CBA, are cbmpleiri&ata?;^^^ 
aspects of the ' same ' problem. , '■*■ . ' '^'^r*^^^!^,. 

The effective' use of cost-benef rt ^amalysis depend^y:. 
therefore upon a clear conception of the place of .trai>r.-|: 
ning within the various domains of policy^. Despite ^ 
weaknesses the power of the technique ^^o' assist diec^-^-'Ons 
about investing resources is already cons2Lderable aTid ^ 
too great to waste where limited .respui?ces are;^sabject . 
to so mai^y competing .demands . 

2 - GBA - Techniques and, probl-ems . ^ ■ 

"Part (b.) is a comprehensive description oT'^ 
cqst benefit study of a vocational trainj^ng programme in 
Ireland. ..ft is of particular interest in*4,discussion of 
the expe^rimental design -used (see Chstptep 3,Part Cf ) on 
research methodology). The test group used for the stu- 
•dy are those, wfio receive training, whether or not they 
complete the course or whether or not they Utilise the 
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■ • ■ . , . • . ■ . e 

»V . . . * 

',■ ' ■ ^\ ■ - • 

training-: immediately , af4:erwards (This is Ibecause some 

wastage is regarded as a normal part of aj training 

course). Tl>e coflxrol group are those whof are trainable, 

are willing to be trained J^ut who are' not trained due 

to a shortage, .of places 'in the training centres. 

Section (c ) discusses the method-ologjf^ and concepts of 
CBA and reviews the results obtained so far in applying 
it S^o|- vocational training. Tha paper makes- th# very 
important point that CBA resul's are very sensitive to 
ythe measurement' o^ what are called displ^ement effects 
'i.e. the process Wnereby Jrainees obtarn JiQbs which cQuld 
have b^n ?illed by untrained people and' ^cuum effects 
i.e. the process whereby other members of ^he. labour 
force may be taken, into jobs vacated by . trainees or 
'complementary to those newly occupied by the trainees. 
These effects are extremely difficult" to measure j^d 
therefor^e c^re must -be tak.en in the interpretati(/~of 
results. , ^ 
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(b)' ^ST -^BENEFIT ANALYSIS Qy^NCO DIRECT 
TRAINING PROGRAMMES 

This... is an -pjitline of a study which 'is currently 
being underteken by the Research and Planning Divisionof 
AN'T^O-The' Irish "INDUSTRIAL TRAINING AUTHORITY".^ 

V 

^ The object of the study is the evaluation of the 
costs and benefits ^of AnCO^s Direct Training Programme' 
and the purpose is to find out the returns on an invest- 
.ment in Training in this area of AnCO's activity. 

DESCRIPTION OF DIRECT TRAINING ACTIVITIES .^ 

(a) • Introdja^tion - ' ^ . . ^ 

Under the Industrial Training Act passed bythe^. ^ 
Irish Parliament ,ii} 1967, AnCO - The Industrial y 
Training Authority has responsibility 'for training 
at all levels ^of Indus-try and Commerce. The ^4^ct 
'Traij/ning Division of AnCO provides 'facilities for 
the trainiit^ of unemployed and r^etundant workers ; 
of workers leayi^ng the land, and skill" upgrading 
of ernrployed workers. There are at present nine * 
pern^Tnerit ar^ three temporary training centres 
with a total of 2,040 places and an annual through- ■ 
put in the region of ■5>500 adults^, .1^ . 

(b) Training^Courses Provided - ' ■ ^ 

A wide range of ..courses is provided at the training ^ 
centres. The courses are designed by a central ' 
- ,cour3e design unit and relate as -near as possible to 
''the, needs of i^idustrY in the operative areas. The 
following coifrses of 5 months duration or more are 
'•available forVdults : ni ; . 



-^v- Machine Operating : Capstan and Centre ' 

' Lathe Operators ^ 26 weekg 



Milling Machine 

Operators 26 weeks 

Grinding Machine , , 

Operators 2Q, weeks 
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Engineering Assembly : 



Welding : 



Electronics 



Machine Operating / 26 weeks 

General Engineering p 
Operators 

Domestic Appliance 
Servicing 



^ Welding Operative 
^ Fabrication 

Pan'el Beating 
Sheet/Metal Fa^ricat . 

Elen3tr ical/Electronic 
Assef^bly 

Radio/^.V. Servicing 
Electrj)nics 
Wiremen 




26 weeks 

26 weeks 

^18 we^s ^ 
18 weeks'^ 
24 weeks 
26 weeks 

26 weeks ^ 
56 weeks 
24 w^eeks 
24 weeks 



( c ) Outline of Administrative and Operational Structures 

Each trailing centre is headed by a ■ MarBgor who is 
supported by technical (instructors) and aOKlinis- 
trative staff. The techRjLcal and manageme^fitNger- 
vices are co-ordinated from^the ^headquarters of 
AnCO in -Dublin. These include, on the technical 
side, . Curri'culum Development and Research and 
Planni'rlg 'activities ; on the administrative side. 
Budgeting and Finance, Data Processing and General 
. Administration activities.' 



(d) Cost 



The cost whi'ch is expected to >fee incurred in 1975 ■ 
for the training of 6,59*0 adults is £ 5,605,456 
mO^nCO has made an application to the European 
Sjocia^ Fund for 50 % of these estimated costs under 
Vrticies 4 and 5 of ^the Fund. 



DdKriyptioQ of AdLjjrt Training Characteristics 

The follov^i^g' tables illustrate the employment 
status, mariraJ/status , educational qualifications 
and age of adult trainees. j. 
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ERIC 



Table I 



Table 2 



' Table 3 



Table 4 




Primary 
intermedi^^ 
^Group Cert^^. 
Leaving 
Other 




^ ) Payments to- Trainees 



Trainees._are paid, an al lowance ^ 'ie^^j^ding^°J^JS6 



— are 
home. Travelling allowances 

cases and social welfare creo 



,rnmo<: 



number of dependants (Ta-ble Acc^.^g 
subsidised for" trainees who n ].iy 



in 



away ^ 
certa:^ 



^^ai^^^ovidea^''^ 
^re ^ 



is 
)?orTi 



\ 
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These a i'lowance::; are not subject to income tax , 
T^ble 5 o Allowances paid to Trainees 



. Males and' Ferlales 
With no Dependants 

• Age 18 ' 

Agev 19 

Agtf 20 

'Age 21 to. 55 

Age 55 '^nq ovur' /' 

With Adult pependants 

For each adult dependant; 



Child Dependan^ 

For 1 to 5 children 

inclusive 

"For 4 or more chitdren 



Training .Allowance ' 



£ 10.50 
£ 15.00 • \ 
£ 1^00 
£ 17.50 
£ 19.50 . 

£ 4,00' 



'£ 
£ 



1.00 
2.00 



focial Welfare Paymehts • ' * '.^ '■ ■ . ... 

In cases where Pay Related Socia-l Kelfa^f'Senefit 
in excess of the appropriate training allow$.|j^^,;^^'i*s^ ' 
payable, allowances equal to the amount of ^'e^^l't* may 
be paid. 
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(g) Selectic^n, Placement and Follow-Up Procedures 



All .'trai'hees are aptitude tested bex'ore entering 
training. About eighty per cent of applicants are 
;f ound suitable for the present range of ^.G^u^rses . 
Thbre is a high withdrawal rate prior 'to training 
due to the long waiting period. It .is expected 
that when greater capacity is available in the 
training centres the-wai^ng list' will decline. 

Trainees are given every encouragement to look. for 
suitable job openings while the AnQO centres, which 
they ca;*n avail of uphn f inish^^r^g;,^^^J:i^1^R,^pourse . The- 
Mational Manpower Servi ce, ^through their numerous 
contacts with industry, give considerable help in-" 
'this regard. Over 75 % of the adults traYned in the 
past year ho . e been placed in jobs. In cAses where 
the trainee :in.:3 employment before compiletion of 
the course. 



special emphacl.. is put on the development of those 
skills, which h^ will be "likely to .need in|liis,new 
employnfent . ^' ' ' "^ ' .\. ^' 



OBJECTIVES ^ ■ ' ■ 

Training programmes usually have multiple and 
complex objectives -and this is especially true of trai- 
ning programmes on the national level Where the obj-ecti- 
ves are of ten 'part ^f trfe ^ Government ' s macro-economic 
strategy. The objectives**of AnCO^s adult training pro- 
gramme can be summarised und^^r three-' headings : 

Efficient Allocation, of ^ Resounces ' ' • 

'This iriQluaes the. objectives, of - long run icogiomic 
growth ,,ana i:he corr^'^tion of structural imbaja^^^s in- 
the e o T i o my . ' r / [•» ' 

|- Dis'tributiona-i Equity , , o-^r. . ^ 

' ' Thi^: irjcludes the reduction oi*,. poverty and the * 
pro:/i%ion -of grea.t-eV 'S^eeurity of employment. 

" E c<pj3.o1jtjg^ S'tj'^-i 1 i s a t i on ' ..^ ' 

.•■*■-. "This inc.ludeis- _f he cutting' of unemployment and 'the 
rre-duGtion of inf fatioriary press6ires.. ' ~ - 



Depending^on the. -iecpnomic ' climate prevailing in a 
country one qr other •^*<gr>^the. objejj.tives mky .receive grea- 
ter :em^hasis but j-t is*^impo3sible to take.,%tipn to ^ 
achieve a^y of them without affecting all. th^^" objectives . . 
For example, programmes to cut unemployment :iLUtomatically 
effect the allocation of resources and h4V^^^~- 'in^P^ct 'on 
inflationary pressures. ' - ; 

These objectives are not unique* to 'national trainj.ng 
programmes ; they apply equally to -most Government'.pro- 
je-ct^. ' Thi's facilitates comparison find out which 
project most successfully fulfils the objectiv.es. 
However , 'it . is not «asy to gaugfe the success of projects 
because some objectives are I'ong term and diffuse^. Eva- 
luation methods therefore, tend to be. partial and' 
incomplete. It is i-n the area of externalities - where 
benefits accrue to third parties and not to indivldiAals 
and firms direct ly.' involved - that Government interven- 
tion is., most .justified but it is in this area that 
'measurement is most difficult. 

a^- 1 ^ ^ 



COSTS AND BENEFITS 



The costs and benefits of a training prograrimie , 
are normally delineated fr.om three points of view. 
Society, the Government and the Individual. Before 
doi'ng this it is. necessary to make ^a distinction bet- 
ween the impact of training on the actual output of the^ n 
economy ah5 the impact on an economy's productive* 
capacity. ^The costs .arid benefits vary depending on whi<3h 
impact one is' tryinig to .measure : if one wishes to mea- 
sure the impact, on actual output, then vacuum, displace- 
ment and multiplier effects (see below) shpuld be 
included ; if one wishes to measure the impact on produc- 
tive capacity, no^ allowance should be made for multiplier 
effects, and the earnings forgone during training reflec- 
ting the reduction in productive capacity of putting 
'people in training should be regarded as a cost whether 
or not they have been repl^'ced by other workers , Mefelly 
the impact should be ca.lculated'-f rom both the productive 
capacity and actual output orientation and this will be 
attemptied in the study. A distinction will; be. made in 
the results between hard facts and soft as'sumptions - . 
because^ of the difficulties associated with measuring 
vacuum and displacement effects. . . ♦ 

Benefits to Society 

The benefits to society 'from training are } • 

. (l) -The . increase in production after training and ..as 
a result of it. (Since the increase in production 
cannot be measured directly the .irfcrease in the 
trainees' gross earnings will be used as an indi- 
rect measure of the increase in production)!^ 

(2) Value of work done by trainees ^during the training 
course . • * 

(3) Ext^a new jobs and the vacuum effects 'associated 
with the new jobs. (An attempt will be made to 
estimate -^hese .from a study of labour market condi- 
tions prevailing during the course of the evalua- 
tion). 



Footnotes - Vacuum Effects refer to the process whereby 
other members of . the labour force may be taken into 
jobs. vacated by trainees or complementary to tnose 
newly o*ccupied by the trainees. r 
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(4) Socio-psychological benefits i.e. increased Job 
^ satisfaction, reduction in crime; etc. 

Cost-s to Society . » , 

The costs to society from t rflning are : 

(1) Production f orgoneNiuring training. (The reduction' 
'in trainee gross earnings while training will be 
used as an indirect measure of production forgone). 

(2) Capital costs. (This covers the opportunity costs 
of buildings and equipment). 

(5) The costs of materials, administration, and wages 
and salaries paid to personnel. 

(4) Socio-psychological costs. 

Benefits to Government . ♦ 
The benefits to ' Government are> : 

r 

'(l) Increase in tax revenue as a result of higher 
earnings . 

(2) Saving on social payments as a result of increased 
employment. 

(5) Value of work done byr trainees during training. 
Costs to Government 

'The costs to Government are \ 

(1) Loss of tax revenue as a result of decreased ear- 
■ ^ nings during training. 

(2) Allowances, travel and subsistence^ payments 'to 
^ trainees. . ' ^ ^ 



Footnotes (Contd. ) Displacement effects refers .to the 
process whereby trainees obtain Jobs which could have 
been filled by untrained people. 

Multiplier effects refers to the process by which an 
increase in investment generates extra income. 
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(3) The opportunity costs of expenditure on building 
and equipment . 



The benefits to individuals include 

(1) Increase in after-tax earnings. 

(2) Socio-psychological benefits i.e. increased job 
satisfaction, etc., 

Costs to Individual - . 

The costs, to individuals include 

(1) Net cEtthings forgone during training. 

(2) Socio-psychological costs. 

EXPERIMEMTAL DESIGN / • ^ ■ ■ 

Measui^ng the impact of a -training programme is 
essentially the problem of calculating the difference 
in variables betv;een t\^^CT\simultaneous yet mutually* 
exclusive states, when the programme exists and' when 
it does not exist. A number of experimental designs 
with a-* control and test group comparison can be used to 
isolate, the impact of training. 

1. Before - after^ comparison ; the person s record 

^ before training is compared with his record after 
training (in other words the pre-training experien- 
ce is used as the control). (Scott) 

2. . Contfidl groups from the target population - the 

conWol group is selected from people who appear 
to haSt^ a similar background and characteristics to 
the test group.' (Page, Gooding,- Cain and Stromsdor- 
fer, Solie, Gobbard and Somers ) • 

3. Control groups from dropouts. (Borus) ^ 

4. Control groups from qualified int^erest non-enrol- 
lees - persons who apply but do not enroll for 
various reasons (Hardin and Borus) 



(4) The cost of materials, administ 
and salaries paid to personnel . 

Benefits to Individual^ ? 




5. Cbntrol groups by "Snowball" technique. This type 
of control group is obtained by' canvassing acquain- 
tances nominated by the trainees to find people of 
a similar background and characteristics. (Main) 

6. Control group by random assignment of qualified in- 
terest persons. Applicants are randomly assigned 
to either a control' or test group. 

•In choosing any experime'ntal design it is necessary- 
to satisfy the conditions of internal and external vali- 
dity. (Campbell and Stanley). Internal validity is the 
basic minimum without which any experiment is uninter- 
pretable. It guarantees that all exogenous factors are 
controlled and that changes in the': dependent variables 
can be "clearly discerned and opportioned to the inde- 
pendent variables, otherwise a number of effects are 
compounded" and no clear-cut interpretation of the effect 
of training (in this instance) can be made. 

External validity guarantees ■ the representativeness 
of '^he experiment t to what populations, settings, treat- 
ment ^riabies and measurement variables ' can the effect 
be generalised.^ Exterrtal validity is an ideal to be 
aimed at rather^ than a goal which can be easily realised. 

Designs -1 - 5 are^ op^n to object ions in ,yaryirlg 
degrees' on the grounds of inte'rnal and/or external 
^validity. In these designs mul^tiple regression techni- 
ques can be used to isolate the: effects of exogenous 
factors but in a non-r£[ndo.m situation the competency 
of these techniques depends upon the inclusion of- the 
variables in tables or eqifiN:ions in corfect mathemat icial 
form'. > In contra.st, using Design 6, "Control group by 
random assignment of qualified interest persons", the 
treatment effect (training) can be measured direct ly, 
once randomisation has been achieved^by the ' difference 
between the mean, values of the dependent variable for 
the treatment and control group. Design 6 satisfies the 
conditioas- set by' internal validity and this is the type 
of djesign chosen for the AnCO study. Design 2 appears to 
satisfy the conditions set by external validity but if 
the control group is chosen from the target population, 
rather than applicants, then the effect being measured is 
not purely training but the compounded effect of motiva-\ 
tion and training. " ^ \ 
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EXPERIMENTAL DESIGN AnCO STUDY. 



The following are the test arid control groups for the 
AnCO study ■: . ^ ' ' . ^ . 

1. Test Group" - ' ; . ' 

• Those who receive training, wh^ether or not 'they 
complete the course, or -whethfer or^not they^utilise 
the- training immediately.;;^ afterwards . (Those who. do 
not complete training or'-utilise^ it afterwards are 
included because sbme wastage is' regarded as a 
normal part of a training course).* 
- ,' ^ ^ ■ ' (/ 

2. Control Group 'j^ ' - 

Those who are trainable, are ^T?^illing^ to be trained, 
but who are not trained due to a shortage of places 
in the training centres. 

Type • of Sample ' ■ / r ■ ■ 

A stratified random sample is being chosen because 

a) it' is reasonable to hypothesise that some sections of 
the trainee groap will be more variable than others 

b) it' is necessary to ensure that small categories are 
adequately represented. 

Possible Criteria for Stratification 

There are a number of possible criteria for strati- 
fication. . 

Age ^ ' 

It is reasonable to assume that age ip an important 
factor which determines the returns to training in^erms 
.of the pay-off period and abilityto learnyiew skills. 

Employment Experience \ ^ cS^ 

The employment experience of the trainee prior to 
training is a factor which .cguld also influence the • 
trainee^s post-traininfe experience. 
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r • ' * ' •< • ' 

Educational Experience " _ . ^ ; 

The educational experience of- the trainee is a 
factor which could influence .the 'trainee Vs ability, to. 
acquire new skills and find employment subsequently. 

Course ^ ' . . , 

, ' The. training course is an important determinant of 
'the returns to training in terms of amount invested and 
the trainee ^s success in Mnding, fenjploym^nt . 

■^^ • ''' ^ ' L 

Region - 

Regional , differences^ in labour market conditions 
are an important determinant of the returns to training. 

Marital Status 

The marital -status of the trainee could also be an 
important influence. 

In the absence of f io:^nieyidence to support the 
effects of such factors on tfe^ employment experience of 
those who pass through -the training centres, the most 
appropriate method is to stratify by. factors which are 
easily identifiable and more objective in nature, i.e. 
age and training course. The, most practical me-thod of 
selection of the sample is on a training centre basis 
.and this will ihdroduce further stratification*. The 
effects 'of other factors will be 'regarded as "domains" 
to be studied rather than criteria for stratification. 

Method 'of Selection of Sample * ^ . 

("a) Compile a list of applicants considered suitable for 
training in each time period. Divide these into 
strata^ classifications . 

(b) Choose the required proportion at random from each 
stratum list and assign them to be trained as .the 
test group. 

(c) Choose another random group the same way and ensure 
- • that^ -they -are -no t- -trained-f or- the - d-uratioru .of. '^the- ^ • • - 

study and -regard them as the test group. 

(d) Those not in either group to be treated in the . 
normal way. • - " 
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(e) Continue this procedure each time .period until a 
siffficient number of people have been selected,- 
A preliminary analysis of some of the earlier peo- 
ple nLay_^ .indicate the_ _op t ima 1 _o ve ral l_s amp^^^^^ , ji.^^. .v.^l 

DATA COLLECTION AND INTERVIEWING PROCEDURE 

The data will be, collected by direct interview with 
members of both groups. The interview will cover such 
areas as . : ^ 

Personal Vitae 

Educational Background 

Work Experience 

Income 

ilea 1th 

AnCO Training / 

\ 

There will be 'four interviews , with each respondent, 
the, first when the test and control group are completed, 
the second when the test group complete the training 
course, the third six iponths later, and the fourth 
interview will take place eighteen months after the 
AnCO training course .has been completed. 

RESEARCH CONTRIBUTION ' 

Einar Hardin in a review of research on the economic 
benefits and costs of training programmes, fias offered 
the following recommendations among others, as to how 
research in this area might be improved : 

. I 

1. "More emphasis should be placed on estimating the 

variations in the benefit-cost relationship asso- 
ciated with forms and degrees cs^f training and 
other conditioning variables". 

2. "The follow-up period should be lengthened". 

3. "Methods of strict random sampling should tje substi- 

tuted for' the judgment sampling of more published 
work" . - ' ' 

- It isv intended that this study will make a contri- 
bution to research in each of these areas. 
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(c) COST BENEFIT ANALYSIS OF TRAINING Pftt)GRAMIvE^S ^ 
Introduction _ ' 



• 



This paper has two objectives. First to. make the 
wqrk of economists in this field more accessible to those 
administrators and professional advisors who are not 
already familiar with it: Secondly, to inelicate how the 
results of existing work - and hopefully future research 
work- can be'utilised in the formulation of policies, 
for training. The paper makes no pretensions to be a 
techni^cal exposition ; it seeks only to explain enough 
of the concepts "and technical jargon to give the method- 
ology and results of economic analysis a wider audience 
among those concerned with evaluation t^i:^aining program- 
mes. Part I provides a brief account of the nature and 
value of cost benefit analysis (CBA)* ;^Part II discusses 
some' methodological problems ^and Part III sets out some 
results and di's cusses some implications of research 
work for policy analysis. 

" Part I % 

■ Cost benefit analysis (CBA.). is a technique^ of in- 
vestment appraisal. It seeks tc^ identify ^nd estimate 
all costs and all benefits associated with any given 
programme of expenditure and to evaluate the return on 
that expenditure. The results can then be compared with 
the returns available on alternative programmes of ex-, 
penditure and thus one can' ensure an efficient use of 
scarce resources. If returns^can be calculated - either 
as rates of return on capital investment or a^'iTet pre- 
sent values of disco^^ted future annual benefits (2), 
then CBA provides a clear decision rule to policy makers. 



, Any comments made in the course of this paper and sub- 
sequent discussion are personal and do not necessa- 
rily reflect those -of the UK Training Services Agency.- 

, Money is worth more' in the present, than in the future 
because it can be useS productively, benefit's occur- 
ring over a future time period must therefore be dis- 
counted at an appropriate rate of interest to give a 
"present value" ^which can be^ compared with the prop.osed 
' " i h V e^*t m e n t" "'f p r e s e h"t" "cost)!"" " 'V ■ 



130 



^31 



'lt*;indicates' that high or'low re-turns can ^xpec-. 
ted for.. various' options or for competing policies and ^ 
hence offers a clear guide to decision makers on the 
economia^'^spects of policy decisions. It is well reco- 
gnised that not all decisions are made on economic cri-- 
t^ria b'i^-^ it is -pre sumabiy accepted that econamic consi-"' 
deraliion^ are an/ important element in most decisions • 
involving resources. — ^_ 

A rate of return does not offer a guide to the 
question ^^hjpw much training, should be financed /provided , 
by public 'agencies ?" li indicates 'only; that past invest- 
ment has generated a return of and the size of x is 
an indication of the directiftn of investment (more, less, 
the same) '"but not the size of any- increment Of investment, 
although Clearly the size of x does suggest the rough 
order of magnitude of any further investment. 

CBA is perhaps a more useful evaluation tool for the 
planner and policy maker than the, professional .trainer, 
for it is concerned more with the return on resources 
devoted to training programmes than the internarl benefits 
of a traifting package such as the acquisition of skills • 
etc. However in both cases there is an interest in the 
return on resources, and this is what CBA seeks' .-tO; . eva- 
luate.- 

Although the methodological difficulties of, achie- 
vi-ng such measures are discussed in more detail later it 
is clear* that money values are required for the cal- 
culation of any rate of return or any net present value. 
F.or various reasons this is not always practicable and 
the appropriate technique then becomes cost effectiveness 
analysis. . . . v ' 

Although-^ unlike CBA, it does not- Offer a decision 
rule the results of cost effectiveness analysis are ©ften 
strikirf^tg enough to allow recommendations, of one . policy 
rather '^^than another, and at, the very least one is able 
to attach costs to "policies and to have some notion 
of along what dimensions policies may be effective once 
implemented. The , technique contributes to policy making 
in two main ways,^ i ,^ , ' ^ ^t!^,.. . 

, ^ ^ ■ . s 

(I) where alternative policies with' the ' same objectives 
are being analysed it is sometimes the case that 
there is no. significant difference in the effect-i^ — — 
veness of policy A or policy B in securing a defined 
objective x. ' In this case a fairly straight forward"" 
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cost minimization analysis is, possible. 

(II) where effectiveness measure are different then one 

can show the tradeoffs along the , various dimensions- 
of .effectiveness. Thus one training. ^programme may 
lead to training of a higher quality (at a given 
cost) but another policy may generate more trainees 
In a given period and at 'a given cost to receive 
"a somewhat lower level of trainiTig, The trade off 
.is then one of time span, quality and numbers at a 
"given cost such 'that -sajy '^1 million invested in 
policy A gives x benefits per unit of time and the 
•same sum x givesx y benefits per unit of time if 
invested in policy B. The final decision of course 
lies with the planners and policy makers. 

Thus either one constrains objectives and effecti- 
veness measure and minimises costs or^one establishes 
the trade off between costs for .given' achievements. 

There are- two other small points to add. First 
there may be occasions when the mere setting out ,of costs 
and probable effectiveness of two very, different policies 
will give such striking results that this alone is a 
helpful guide to decision making. Secondly, , one may 
adjust .the technique to take account of decisions 
between major policies, and decisions between options, , 
but within an' agreed . programme . Thus having dedided to 
support retraining policies for the disabled (rather than 
immigrants, young persons, or women etc) there still 
remains the choice of the best option within the disabled 
programme. Cost effectiveness analysis is applicable, in 
principle, Ao both kinds of decision. 

In short then CBA and cost effectiveness analysis 
are techniques of programmes appraisal. The results 
can be directly utilized in-decisibn making or policy- • 
formation". CBA does provide a decision rule but it 
indicates only the direction of investment rather than 
the 'size of any increment in investment 

Part II - Methodological Problems 

• There ^are- relatively few problems of methodology 
where CBA or cost effectiveness analysis is conducted at 
the level of the individual or the level of, the firm. 
While cost .^af a presents rather more difficulties than 
is usually allowed it is nevertheless ©ot a serio-us 
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obstacle to useful analysis. On the ^benefit side indi- 
' viduals seek to achieve employment or ^ more stable em- 
ployment or more satisfactory employment, and of course 
higher earnings. Benefits to the firm are usually . 
captured- in two major measure? - that of increased out- 
put or increased prpf its and of course reduced labour 
turnover. Other benefits to the firm are listed as < 
reduced waste, irjjproved quality, reduced requirement for 
superv-i'sion, lower "over-time costs, reduced requirement 
for mdfi'nt?enance of machines, reduced injuries and acci- 
dents, '"better communications and general raising of 
employee jnoral and labour conditions. Some of these 
benefits can be captured in measured terms but most are 
subsumed under the two measures of increased output and 
reduced labour turnover. It should be^said immediately 
that this kind of analyses has only been successfully 
completed for operative training within the firm. The 
problems of supervisory training and management training 
raise difficulties of an order of magnitude which has not 
yet been successfully analysed, 

.The ma jor' methodological difficulties howeveY^ are 
concerned with the evaluation of training prpgrammes 
with respect to public expenditure,' That' is the costs 
and benefits to society of public intervention in the 
finance and prpvisi^on of training programmes. On the 
cost side most economists, accept that the cost of trai- 
ning should include the foregone earnings (output) of 
those who enter training programmes. This is not rele- 
vant of course 'Where the trainee was previously unem- 
ployed but where Tie enters training from employment then ^ 
clearly society has lost the^ output, for which he was 
previously responsible unless he is replaced immediate^^Ly* 
from among the unemployed. His, earnings represent a 
proxy measure of foregone output and this^ miist be inclu- 
ded on the cost side, of , any CBA. The major difficulties 
of course relate to; measuring benefits. We should be 
clear as to the objectives of" publicly supported training 
programmes. In most countries the objectives are some 
combination of the following : . 

- training facilities which allow individuals to 
acq.uire new skills, craft trades and/or social 
skilly and hence aspire to new job opportunities 
particularly in areas of labour shortage 

- training designed specifically to alleviate * 
unemployment 
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- training- facilities for'certain specially-' 
disadvantaged groups - handicapped etc. 

In all these cases the major benefits 6an be taken 
to be either some net reduction in unemployment or net 
increase in the durabi li ty and stability of employment ^ 
by individuals or increased earnings of individuals 
or an appropriate jnovement of trainees to new job/indus- 
try opportunities., * Measuring these benefits gives Tlse 
to certain problems. 

The basic assumption is that any increase, in output 
in- goods and services due to the productiviJ:y of trainees 
is measured adequately albeit- imperfectly by their 
earnings. The differential between pre and post training 
earnings is central to the evaluation of direct benefits 
of training programmes. To measure earnings is to assume 
"that the man is employed and certainly the employment 
position or post training situation is an important part 
of CBA evaluation. Indirect benefits-what is known in 
the literature as externalities - are also crucial to the 
evaluation. The vacuum or replacement effects occur when 
jobs that would have been held by trainees had they not 
joined the training course are now'filled either directly 
by the unemployed or are take^ by other employed workers 
' whose vacated job in turn, creates a chain reaction 
through the labour market leading eventually to the em- 
ployment of hitherto unemployed workers. The effectr of 
replacement iay'to raise benefits and lower costs.' 

Displacement effects occurs when .tra^inees on comple- 
tion enter Jobs thaf are held by others or which would 
have been filled by others without tnaining. Training 
lead's not to a net decrease in unemployment but simply, . 
a reshuffling of the unemployed. The circumstances in 
which displacement is likely to be severe are not easy . 
to specify. In a -shortage - and an increasing shoi**s^e - 
situation the displacement effect is likely to be small. 
In an oversupply - and rising surplus - situation the 
effect is more severe. The displacement effect is likely 
to be more severe where trainees find employment in ^ 
occupations- other*, than those of the training trade. 

The- multipller. ef f e,ct. ..Q.ccur.s when traipses enter 

occupations which currently are in short s'upply. Bottle- 
necks, i.e. constraints in production work due to a 
labour shortage, are now released through this increase 
in the supply of appropriate 'labour. It sometirrips hap- 
pens that labour - say skilled and Unskilled js j 
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complementary and the provision of. a ski^^ed man may 
lead to the employment of an ui^skilled as-sistant. 



Thus an expansion of out/plft is now possible because 
a bottleneck is released, causing some net increase in 
•employment (reduction in unemployment) due to the joint 
demand of skilled or unskilled labour.. However there is 
a real problem in measuring^ that proportion of increase 
in output due solely to training.^ 

There is soine disagreement .within the .literature as 
to the way these .technical' issues m^y be resolved a-nd the 
usualy way forward is to. offer a range of values which 
will illustrate the impact pf these effects on the - 
results. One thus has some 'idea how . sensitive any CBA 
results are to these effects. 

^ Macro economic benefits are recognized in|?the lite- 
rat urFTuFTreseTdoFTstima ted in empirical work due to- 
the great difficulty in idefiti f icat ion and measurement. . 
The argument is that if training is concentrated in .tho- 
se areas in which there is a current la'fcour shortage and 
if the employment service is geared to increasethe 
'efficiency in job matching so it becomes P9ssible to 
achieve counter-inflation policies without incUx^ing 
high l^els of unemployment or reflationary poli'cU-es 
without causing severe inflation. The difficulty of 
estlmaUng suQh benefits usually prevents any empirical 
work in this field. There is however some discussion of 
the policy implications for training programmes. Should 
public training programmes be expanded during periods of 
recession and contracted during period's of boom or vice 
versa ? Should the composition of such training pro- 
grammes be changed such that training for skills take 
place during recession thus enabling skilled labour to 
enter' labour markets a'pd reduce bottlenecks in the sub- 
sequent period of recession and training f or.%^o.,cial 
skills take priority during period of boom ?. "'' ■ 

.Another issue of major importance is the redistribu- 
tive effects of training programmes. CBA. is essentially 
concerned' with the "efficiency of investment decisions. 
It is usually assumed that current income distribution is 
socially^ acceptable and that where redistributive effects 
of investments- do -occur they can- -be ^ captured^ by . treating 
each persons gains and losses equally. In many cases 
the redistributive effects are likely to beso small .the 
question can be disregarded. 
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However, ■ society may wish to alter the distribution 
of income between the employed and unemployed ; between, 
colour, classes, and races ; and perhaps between yoUng 
and old within society.^ Thus, some persons'" gains or 
"losses are now regarded as^more important than others, . 
Manpower retraining policie^s .are sometimes the instru- 
ment of" such social objectives, Certainly training may 
'be regarded as an investment In humai^ capital, and as 
human capital generates an income stream, changing the 
distribution of human capital -shoi^ld affect the subseir. a 
qiieVit distribution of income. How far human capital 
investment determines (pre tax) inc'bme distribution in 
society is not clear but, either deliberately or ■ inad- 
•yeptently, it is likely to have some impact and if we 
■^•%*'&^^^nable to quantify the effects then CBA study of man- 
power • programmes is likely to be incomplete. While it 
is the case that most of. these' issues still require 
further analysis and empirical testing there must be 
y/^few who do not consider them to.be important aspects of 
' 'cost^benef it analysis in the evaluatldh of training. 

Part III Re suits 

•Unfortunately I am familiar only v^ith the United 
States and UK material and one Swedish study ; I would 
welcome references to European and other work. 

At the. company level iPr6fessor Brinley Thomas and 
Dr. Alan Jones and their associlates hav§ found that 
investment in operative training gives a benefit cost 
ratio of some 6- to 1. Similar results have been found 
whfen the studies were extended to similar training in ■ : 
similar firms but in different industries Dr.Zidermah 
of'London University in his analysis of government pro- 
yided training found high and. positive returns, which, 
depending the assumptions made -regarding external bene- 
fits, are of the order of 10 to 15^, using a resultat, 
^"^•nyi^stment costs were recouped within five years. 

\ ' In- the US work, by Page. on 900 trainees in Massachu- 
^s-etts between 1958 and l'9ol under the Manpower Develop- 
Vient Act (MDA) found net benefits in excess of £3m. 
That is, the money value of benefits of, training received, 
less costs, exceed £5m\ Analysis of the Job^J^jrand , 
^"NeighborhcS^d Youth Corp programmes- both of ^W^ich relate ■ 
■ to young persons suggest .mixed succesp. Byit, given high 
costs and difficult training material, '.the post^^t raining 
employment and earnings data is 'impressive - CBa ratios 
of "2;1 or 5:1, depending on assumptions', are reported 
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in Thurow. ' ' " . 

Borus calculated Benefit - cost ratios for Connec- 
ticut training courses ranging fr.om Jil to 6:1 depending 
on assumptions concerning the use of skills, over ^a 
lOyr service life. Cain & Stromsdorfer calculated 
ratios of 10:1 for men nnd 2,-7: 1 for women, and an ove- 
rall ratio of 9:1 in a sample of West Virginia training 
courses- ^ Using the same data • Stromsdorfer calculates . 
rates of return of 90 ^.for men and 64 % fof females. 
Hardia & Borus - calculated a B/C ratio of 1:2 for a 
sample of Michigan trainees , using a lOyr life and a 
10 % discount rate. 

In Sweden Dahlberg assumes a l.Oyr life v/ith a 
decreaSlfig benefit stream. Discounting at 10 % he 
calciilated a B-C ratio to individuals of be-tween 3.2 
& 4.C ^nd a payback of 21-55 months. The social B/C 
ratio ■ depending on assumptions lies between 4.5 ^ind 6.7 
with a payback period of 17-20 months. 

What are the implications of these results for the 
evaluation of publicly supported training programmes ? 

, First, all results suggest that some expansion of 
training facilities would be economically justified al- 
though the scale and composition of such expansion raises 
other issues.* 

Second , some close empirical estimates of the dis- 
placement and replacement effect are needed because CBA ■ 
results are quite sensitive to these factors. 'Economic 
analysis 'is less than conclusive about the-benefits of 
counter cyclical training pnogrammes and this deserves 
closer consideration as it is becoming a major policy 
question in many ind-u'strial counties. Finally the re- 
quest for more CBA research work, 9n different types of 
training courses, on different regions, and on different ^ 
age and sex groups, is no cliche, hut represents a badly 
needed programme of work -if policieW" are/to be more care- 
fully formulated and implemented."^ However despite the 
imperfections of current work- the results have TDeen inte- 
resting and valuable and I hope due consideration will be 
given to economic appraisal by those of you not already 
familiar with the methods and results of CBA for the 
evaluation of training programme. . ' • ^ 
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CHAPTER 5 



e'Ualuatiqn and ITS ' /application 



Underlying much of the discussion in> the semindr 
was a desire for a more integrated approach to evalua- 
tion. At present, a decision to invest in a particular 
programme of training is likely to be taken in' the light 
only of th.e most immediate employment outcomes. Yet 
questions concerning the manpov/fer system as a whole are 
inevitably raised by such a decision. There may also 
be even wider 6onsequences of a social nature. Training 
may produce not only skills, not only employed men, but 
also unemployed , men (displacement eff ect) , a demand for 
further training, geographical mobility and so on. The 
effect of one particular training programme may be 
difficult to see, but the national sponsors of training 
canRot treat individual programmes just as isolated 
entities. They are inevitably seen in the light of a 
policy and a policy must be based on aggregate effects. 

The process also works in reverse. Changes in the 
larger scene must be met by an adequate response from 
the training system. The examples of job-enrichment, 
the apparently ever-growing need. for versatility or the 
ideology of industrial democracy constitute clear demands 
on the training system. To meet such demands 'adequately, 
it is not enough to respond in a local fashion : the 
global policy of the training system must anticipate the 
local expression of needs. Without such a glqbal res- 
ponse, ^ the resources of training expertise and financial 
support are likely to be out of phase with demands as 
they arise . 

In the language of the model presented earlier, the 
policies of. the various domains need to be linked cohe- 
rently. It:is.the objective of integrated evaluation to 
achieve this," for in. practical terms the effective stimu- 
lus^to policy integration is the clear demonstration of 
the nature of the . dependencies . To achieve this the eva- 
luation of training must start to incl»ude the bigger 
question. ' - 
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This does not imply that the smaller more de/tailed 
ones are replaced, but rather that they are placed in the 
•larger context more explicitly. Studies of labour market 
structure , of underlying personal competencies implied 
in this structure and of costs and benefits' are'' examples 
of initiatives that are under way ancj which have already 
shown that they can produce results and insights that 
are worth building on. 

Implicit in the integration of levels of out.come is 
«the integration of outcomes through time. The. financing 
of training projects is usually- set 'withifi a fairly short 
time span or one that is arbitrarily determined by 
institutional factors. Bdt outcomes in the various do- 
mains have a time structure of their own. A training 
project which has a significant impact on the manpower 
system (whether at the level of one organisation, regio- . 
nally or nationally ) "may^ bring in its train further needs 
for training after perhaps a year or two. One of the 
working groups set up during the seminar produced a 
simple table showing the time sequence^ for evaluation of 
the outcomes in the various domains (diagram 4). 

The' context of training systems outlined in the 
Seminar supports and elaborates the notion of training 
as the management of change. This has two aspects to 
it : V 

(1) The conceptualisation of the situation of training, 
i.e. how other things affect training decisions and 
are affected by training decisions ; 

(2) understanding the art and technology of training 
processes themselves . 

Evaluation is concerned therefore with analysing 
both means and ends. Even if the ends are "given" by 
authority higher in the system, the evaluation process 
should still analyse those ends. Without such an analy- 
sis the assessment of the means will be bound by the 
perspectives of the higher authority. This would be to 
limit evaluation to' a mere servicing function and to 
ignore the educational power properly inherent in it. 
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Diagram k 



Factors involved in the evalilation of the' 
success of vocational training measures, and 
the time sequence for such e/aluation, 

Time Sequence ' 
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The crosses indicate the time at which the evaluation 
of factor's involved in the assessment of success 
should 'best be made, 
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In terms of the model presented in this report 
. ^^ (Fig. 1) evaluation can be seen in the following terms : 

(i) Analysing the Ends. 

• . This consists of determining how a given objective 
relates to other needs or policies within its own d*omain 
and within any superordinate domain. * 

Thus a proposed objective stated at the level of the ^ 
Training Domain must be related to other Training Domain 
objectives or fieeds ,and to the needs and objectives of 
the manpower Domain and the Social Domain. The process 
is ordered and cumulative in that the first integration 
of objectives etc. should be accomplished before carrying 
out the second integration at the higher level. The 
outcome of the analysis of Ends may be called the defi- 
ned objective". 

(ii) Analysing the means. 

The means consist of possible actions in subordinate 
domains. Those chosen are examined so as to estimate 
their ef f,ecti veness in achieving the '^proposed objective . 
Such an estimation may be predictive, using data from 
past experience -where it is applicable ."^ Or it may be . 
concurrent, utilising data fed back from the actions ^ V 
themselves. Referring again to the. example of an objec- 
tive" at the level of the Training Domain, tiie proposed 
means will consist of actions within the subordinate 
domains of Programme and Teaching. The process of ana- 
lysing ' the means is also sequential and cumulative. The 
relationship of the defined objective is first determined 
with respect to factors in the Programme Domain (i.e. 
.^subj^^cted to^ task: analysis:' etc) and then witli respect to 
factors in the Teaching Domain. The' outcome of Means 
Analysis may be termed a "defined means". 

The- process- of- evaluation - may ...now. -_bja.-xeat.a.ted.„.as _ 

the following steps : 

(i)Tp analyse , a proposed objective by examining it in 
■ conjunction with factors in its own and in superor- 
dinate domains": a sequential, cumulative, "upward" 
process. ^ 

(ii)Thus to arrive at a defined objective. 
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(iii) To examine the defined objective in ^^nj unction 
with the proposed means : a sequential, cumu- 
lative,^ 'Mownwa;?d" process . 

(iv) Thus to arrive at a defined means. 

The outstanding aspects of th^def inition of 
evaluation emerging from the* Seminal are*: 

(i) The need to integrate objectives at different 
revels ' . 

(ii) The need to understand the art and technology 
of training. 

From (i) it is clear that evaluators cannot be 
regarded as only technicians. Their familiarity with 
all levels of policy is vital if the educational effect 
of evaluation is to be maximised. 

This point is re-inforced by the word "art" in 
part (ii). Teaching, -I'earning and training can be 
^lystematised only to a certain .degree. Furthei«ipre , 
the sy^ematisation will lead only to simplisti|^ con- 
ceptions unless it is set in a genuine understanding of 
the underlying processes. ^Models, such as those presen- 
ted here, checklists of key points in training programmes 
etc, only go so far. They point to ,areas foi? exploration 
or definition : they do not offer a substitute for the 
ability to explore and define. 

The abilities indicated are formidable if they are 
assumed to reside within a -single individual. The 
ability to make -j udgements about a learning process, to 
relate a programme to the field :0f training/education 
polic]^ and provision, to set this within total manpower 
requirements, to see t^is as a part of social policy to 
design appropriate enquiry methods within any of the$e 
'Tields ""and::to' apply "^'s^pecia techniques 's"uch''a's' "cost- 

benefit analysis across this spectrum clearly requires 
a team ''of evaluators^ . . How many and what their qualifi- 
cations should b*e will. Vary with circumstances, but in 
general the following fields of expertise would be 
represented t> 

fa) Social administration/sociology 

(b) Labour economics 

(c) Investment appraisal 

(d) Training management ' 
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(e) Training process ' consultancy . 

In practice two or more of these categories may be 
collapsed into one, e.g. (a) (b) ; (b) (c) ; (c) (d) ; 
(d) (e) ....etc. The nature of the problem and^ the 
availabilitybf personnel will determine specific cases. 

It is possible for such teams to be created ad hoc 
and to operate successfully, and this has happened in 
the past. It would be a considerable advantage however 
if such teaijis were to Ije given a degree of permanence 
which would allow team-integration to develop. With. 
explicit training exercises to this end, it Should not 
be hard to. demonstrate improvements such as-^a greater 
degree of inter-subjective consistency between teams. 
At present, there is scarcely any measure of consistency 
in evaluation. Such data as might.be relevant to it, 
such as that relating to interviewers, academic exami- 
ners etc, suggests that the level of consistency is 
probably low. The decisions taken on the' basis of 
evaluation therefcwie must be correspondingly doubtful. 
The use of team .e[xerclses, inter-team exercises, the 
development of w'drking skills and classifications etc.. 
are' an immediate possibility. The Seminar has -shown 
that a framework for evaluation can be offered. But 
the criticism, modification and development of this 
framework can best come from practical attempts to 
implement it. 

One such attempt has 'been made by the Commission . 
of the- European Communities. In 1974 it established an 
international team^^trf advisory experts tp examine a 
number of training prog^rammes supported by the European 
Social Fund. For the first serie.s of projects examined 
they evaluated the extent to which the training met the 
objectives set for it. .In the current series they are 
looking at the relationship between the training pro- 
gramme and the overall manpower and socio-economic goals 
, of,- the- -country- concerned-. - Apar-t-.-from- the . usefulness- of 
the evaluation itself and its importance for the develop 
ment of the Social Fund the work of the team is generb.- 
ting a transnational understanding of the training -"; ■ 
problems of the different member states and is estatrlis- 
hing a fund bP expertise and a measure of tearrt 'integra- 
tion despite the varied backgrounds (both nationally and 
professionally) of its 'members. 
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